A Dream Deferred

This is a hard blog post to write. I’ve actually pondered for some time whether I should write this down and publish it or just bottle it up and deal with it like we did with a similar situation two and a half years ago. I’m not one to shame companies, stir the pot up and make a scene. But I still feel some seething anger about what I perceived as discrimination then and I feel just as angry about this new situation I’m writing about today. After some reflection I’ve decided to simply tell this story and my hope is that, while venting, perhaps find some positive resolution. You may say I’m twisting events or that I’m over-reacting. You may put my wife’s character and capabilities into question. It is what is and sometimes I think it’s worth it to just share a story. To put today’s story into context I think it is best to start from the beginning, talk about the experience I bottled up and then describe today’s scenario. I’ll draw my own conclusion while you may draw your own.

My wife is a very ambitious woman. Born in a refugee camp by two parents who met after losing their loved ones under the Khmer Rouge regime she was born a fighter. She survived a bout of lung cancer when she was only two years old and to this day has only one lung. She graduated college with a Bachelor’s in Mathematic and in Computer Science. When we worked together I was easily attracted to her ambition, intelligence and genuine concern for others. She always gives a twenty dollar bill to the homeless on the street. When she encounters a veteran or military personnel in uniform in public she always thanks them for their service. But I always loved how great she was to work with… we had so much fun attending conferences together and even gave a joint presentation on Behavior Driven Development with Scala at Strange Loop back in 2010.

Where I’m going with this is my wife has very specific career goals. Sure it can be fun to be an engineer but at one point or another one has a natural desire to move up. The employer we both worked at in the past unfortunately had a very flat structure with little upward momentum. So she began hunting and landed an engineering team lead position at another local company. She spent her days mentoring junior developers on practices like test driven development, object oriented design, patterns, etc. She also provided career guidance and even sent one of her developers to management training and put him in charge of some projects to give him experience. As a side story, he went on to become a team lead himself who I hear has been doing a pretty good job. Suffice to say, she really enjoyed this position and often said she was “living the dream.”

Conflict and Discord

Looking back I think sometimes the source of the conflict between Than and her manager was the fact that Than’s ambition is very aggressive and she’s not beyond doing what it takes to get things done. I think an amusing story is when her team had five different product owners say “this is a top priority from our CEO” in different meetings, she got back to her desk, picked up the phone and called the company CEO directly and chatted with him about it and how each five priorities can’t be the top and they need to be prioritized. He really appreciated her bluntness and worked with her to re-prioritize a bit. Her manager gave her a comment that the next time she should go through her.

It probably also didn’t help that her team was very strong willed and unafraid to go against Cargo Cult practices and shake up the status quo. But I always heard good things independently from people on her team. They really looked up to her and respected how much she stood up for them. She never backed down and was always willing to fight for her team when she had to. She never got a bad review and members of the executive team really liked her. Her manager once confided in her that sometimes she felt somewhat intimidated by Than because of her knowledge, experience and aggressiveness.

Something Strange Comes This Way

Than’s team kept growing and at one point was the largest team. I think they had a good ten or twelve people and I know at least two or three had specifically requested a transfer to be on her team. However as other teams began to work on new projects members started being “borrowed” off her team. It began downsizing. Six. Four. Two. This started becoming a little stressful to her. What began as an innocuous “Brian has great knowledge of the LMD subsystem so we’ll loan him out to this project to rewrite LMD” had trickled to no team projects… just support tickets. My wife sensed something was up… perhaps a downsize? She told her manager in her one on one that if there was no reason for her team to exist and they were keeping it place just to let her keep her position they should just disband it. And her manager did with a sigh of relief.

I keep looking back on this incident. Out of six or so teams the only team to be downsized and eventually disbanded was the one with a strong, female, dark skinned Asian. None of the other Caucasian males (who I might add had a lot less experience than my wife does) had this situation closing in on them. However I think I could have accepted if they had used this opportunity to perhaps promote my wife to a position better suited for her. Perhaps the Agile Coach position to transform the department in the same way she had transformed her team? They were looking to add that position and even the CTO mentioned that he thought she’d be a good fit.

In what will go down as perhaps one of the puzzling “promotions” I’ve ever witnessed, she was moved to a Business Analyst position. While this position was dressed as a promotion (after all, her salary remained the same, right?) the truth is that she’d be on the same level as her previous direct report; a business analyst that she successfully trained. I still find this highly puzzling, perhaps her manager thought that she’d be a good fit because she did so well training her business analyst? Her days were pretty much filled with nothing but doing business analyst type tasks and continuing to work the support queue her team used to own (but all by herself).

It should come as no surprise that this was a miserable state of affairs and after about two or three months she turned in her resignation letter. Like I said Than is very ambitious and is not content to what is a dead end. And we could both see that this recent “promotion” was nothing more than a dead end. We moved on. She found an agile coaching position that was located two hours east in St. Louis and gladly took it as an opportunity to move forward in her career.

Over that year I kept feeling anger but remained cool. “Cooler heads always prevail.” is the motto I’ve always lived by. But I cannot help but keep looking back and wondering. Was this what discrimination looks like? She never had a bad review. Everyone on her team thought she was fantastic. Yet at the end of the day she… a dark skinned Asian female who had clawed her way up from poverty, the daughter of elderly refugees… was the only team lead to face a demotion. I was so mad. I wanted to breath smoke. I wanted to grab someone and yell “She deserves better than that. For all her hard work, this just is not how you should treat someone!”

But every time I reflect on the situation, it is a permanent gray area. Maybe it was discrimination. Maybe it was just dumb luck. If we cut it another way, perhaps a team had to be cut and it just had to be hers? Why should another team lead have to suffer just because they were born white and male? There’s also the nagging feeling that of all the team leads, Than as the one who came into conflict the most with her manager. Anyhow this whole situation was topped off with Than’s mother passing away and, three weeks after the funeral, her manager’s response to her resignation letter was “I knew this was coming sooner or later.”

Oh well. As Frank Underwood would say, I don’t cry over what happened in kindergarten.

Cooler Heads Prevail

Than loved her new job. She felt like she was making a difference again. As an engineering coach she put together classes to teach TDD, patterns and all that good stuff. She worked with various teams across multiple data centers to organize their work into backlogs and remove road blocks. She got lots of praise from upper management. Her career was moving again. She even showed me a letter, written by someone who had attended her classes, state that he was able to take what he learned in her class and land a different job with his salary doubled. It was really touching to see how much of a difference she was making in other’s lives.

We did have a difficulty of her living in St. Louis during the week and being home only on the weekend but we worked through it. Richard lived with her in St. Louis while Caitlynn and I spent time together working on her speech issues. We cherished our time together as much as possible on the weekends and, when school schedule would allow it, Caitlynn and I would pack up and drive down to St. Louis and I’d work from T-Rex or a random coffeeshop and meet up with her afterwards. Sure it was hard but we made it work.

Unfortunately the contract took the turn that some contracts just do. The consulting company was bought out by a larger company. The contract’s future was put into doubt. If she remained she might be looking at having to consult on site somewhere very far away. Just to give an idea of where she might wind up, the other coach she worked with was sent to Minnesota. After some clever accounting (more on that one day) we made it so that she could simply take the summer off to rebalance and figure out what she wanted to do next. Before we did this though she did take an interview at another company. As she walked out to her car the recruiter called and told her that they were very impressed. The client would love to build a team around her. When could she start? We discussed it at length and in the end she decided it’d be best to take the summer off so she could refocus. A small sabbatical of sorts.

Summer Revelations

That summer Than found out an interesting revelation that really wasn’t quite a revelation given her entire life to this point. She was just way too ambitious to be a stay at home mom. Now this isn’t meant in any shape to be an attack on those who do chose to stay at home. But she just couldn’t tolerate not bringing income into the household. I kept trying to convince her to perhaps work on a startup idea or something but she just couldn’t focus being in a completely work free situation for the first time in her life. Her parents were too elderly to work and when she turned 18 she was the sole provider of her household. She moved them out of Section Eight housing and into a trailer working at Target while attending college. And after she landed her first job out of college she moved them into their first new construction home at the age of 23. Being without a job and not being the breadwinner of the family was a completely alien experience for her.

At the end of the summer she decided to re-enter the field. As luck would have it, around this time the same recruiter who had tried to hire her before the summer started reached out. “Are you still available?” They still hadn’t filled that position and still felt that Than was the perfect fit. I drove down to St. Louis with her and worked on my laptop in a Starbucks while she met with the recruiter and Director of Engineering. They spoke for about an hour and were all smiles and excitement. As we left Than offered to buy me a nice bottle of scotch to celebrate, she had accepted their offer and was going to head up a Quality Engineering team at this new client. They’d build a team around her and after a trial period convert her to a full time manager.

Life Goes On

The remainder of the year was somewhat uneventful. She loved her new job and her new co-workers. With the salary she was now commanding she decided to simply drive back and forth (a total commute of four hours) every day because moving her career forward was worth it. We did look at houses in St. Louis but in the end just really loved our home and our daughter loved her school and friends. So we stayed put and she continued commuting. The engineering team had some growing pains here and there but they worked though them. Her director became VP of engineering. We attended dinners and bourbon tastings with the product owners she worked with. Life was good.

After some time, her contract had reached the point where she was able to convert over to full time. HR pushed really hard to get her to convert but she kept holding out. She wasn’t sure if she wanted to put roots down or not. We discussed it and in the end she decided well, it has been a nice place to work. It pays well. And they do have that sweet $10,000 signing bonus (repayable if you leave earlier than two years) that could definitely help with our deck restoration project. Plus it was a true management position that she could grow in. After much deliberation she signed the employment offer and returned it. This is where things took a quite distinct turn for the worse.

A New Director

Only a week after she converted to a full employee she received the news. A certain rockstar programmer that her employer had been trying to recruit for some time had finally accepted an offer. He’d take on the VP of Engineering’s previous director role. She was a bit uneasy about this because her VP had told someone else that if he kept up the good work he might transition him into this role and this seemed like a broken promise. She was also a little displeased that the team was 100% uninvolved with his hiring process but after meeting with him for lunch she walked away feeling that this guy was pretty alright.

By the new director’s second day he told Than that he didn’t really see a need for her team and would be disbanding it. She’d convert to an engineer and he told her, straight up, that he thinks she’ll do good being in engineering. When she told him this would be a step backwards quite a bit since the past 4 years she’s been grooming herself for management he told her “Well, if you do good and prove yourself there might be a path forward for some kind of promotion in the future.” When she pulled up in the driveway that evening I met her in the driveway and saw she still had tears in her eyes. She felt cheated. Lied to. To take a management position and have the rug pulled out from under her only three weeks later. Eleven years of engineering experience, four years leading teams and coaching, and to be told in the end that she needs to prove herself. What more does one have to prove?

Again I have a gnawing feeling. I want to push it out of my head, I want to make the excuses. “Flat organization structures are all the rage these days,” I think. Maybe he’s just doing this because that’s his philosophy. Then more circumstantial evidence builds up. None of the other managers are being demoted and my wife feels like she’s being picked on. She’s texted multiple teammates asking if she’s being petty by pushing back against her demotion. They’ve all told her not at all and even added that they’re surprised she’s not fighting back harder.

Are we coming face to face with discrimination again? Am I imagining things? I want to believe that her director is just oblivious to what he’s doing to her. It’s easy to meet someone, so far removed from their engineering origins, and come to a conclusion that they’re not that smart. That they have something to prove. He doesn’t know us. He doesn’t know my wife’s past. He doesn’t know that she co-founded a technical user group with me 6 years ago that is still going strong. He doesn’t know she was a judge at startup weekend that had it’s first place winner actually land $5.5 million in VC funding today. He doesn’t know that she’s very respected amongst so many people in our field.

Earlier today I read a very long blog post about discrimination another woman faced at a company and I began seeing some familiar narratives. Both her director and VP are projecting the perception that she’s the one causing problems. That somehow complaining about being demoted only three weeks after signing on is being “petty”. She’s being singled out as the one who is stirring the pot. Today they made the announcement that she’s transitioning to engineering in two weeks and she went to the bathroom and just cried. She feels so defeated and even asked me if she’s really that bad? Should she just exit this field altogether?

In a lot of ways I’m reminded of a poem by Langston Hughes that I first heard during my freshman year of college that really put the hook in me.

What happens to a dream deferred?

Does it dry up
like a raisin in the sun?
Or fester like a sore—
And then run?
Does it stink like rotten meat?
Or crust and sugar over—
like a syrupy sweet?

Maybe it just sags
like a heavy load.

Or does it explode?

The Road Forward

Sometimes our dreams and career goals have their ups and downs. Sometimes we take two steps forward and then have to take three back. Like I told my wife while calming her, only 4 years ago I found myself exiting a one-on-one with my manager and taking a long walk down the road wondering if maybe I should pursue a different career. Maybe I’m somehow just not cut for this field. Thankfully I lucked out because after that moment a friend’s startup really started to take off and they reached out asking me if I’d like to be their first engineering hire. If I hadn’t been in such a low point of my career I likely would have said no and bypassed what has been the best 4 years of my career.

Back to my wife’s situation, to say my anger has been simmering is an understatement. Yeah I lodge a few angry tweets here and there as I’m wont to do. But being negative really doesn’t solve much. Twitter is full of people who I can look at a long list of complaints of injustice and that’s just all they focus on day in and day out. Sure simmering in anger might feel good but it’s just not constructive in the long run.

I advised her that the situation definitely is toxic and it’s time to move on. But I just wish there was a way to make her get past the feeling of worthlessness. It sticks for awhile. It stuck two years ago and it’s sticking now. Everywhere I look in our industry we talk about diversity. “How can we attract more women to this field?” we ask ourselves curiously.

Like I said, perhaps this is all circumstantial. Perhaps you’ll say my wife just isn’t good at what she does. Maybe you’ll say like her director told her that she needs to prove herself. Maybe you’ll say she’s being petty. I’ve come to my conclusion and my conclusion is sexism. Discrimination. Call me a liar and social justice warrior of sorts if it makes you sleep better at night. I’ve simply chosen to call a spade a spade and will call it out.

We won’t take this lying down though and we won’t go silently into the night. We’re survivors. We’re a team and we don’t stop fighting. I’ll be dedicating the next several days to finding her something even better than what she thought she had with this lousy position. You might think less of me for sitting down and airing all of this but so be it. I stand up for those I love. I also can’t help but think that perhaps just distilling all my feelings about this situation into words will be more constructive than silently bottling it up like last time. I don’t want my daughter to ever have to face situations where it really is debatable if one is facing a demotion for anything other than performance.

These situations have left me with a final thought. In a field that has so many problems attracting and retaining diversity it is hard to imagine why when one is faced with so many set backs that leaving the industry is anything else but a logical conclusion. I don’t mean to be yet another voice beating a dead horse but it’s really time we started doing better.

Tuesday Tech Tip: Create Multiple Directories in One Line

Oftentimes you find yourself wanting to create a directory with multiple subdirectories. For example, when creating Ansible roles I almost typically always create the role and three to five different subdirectories underneath it.

This is actually quite easy to do on any posix compliant shell (although I have only tried bash and zsh).

This will yield the following directory structure.

Pretty simple but incredibly handy. This also works for many different arbitrary shell commands, give it a try!

Replacing Text in Nginx with sub_filter

Sometimes you find yourself in a weird predicament. A third party application that you’ve slapped nginx in front of insists on using internal IP addresses or ports despite your reverse proxy passing all the correct headers and other pieces required. Or maybe you’ve found yourself in the situation I found myself in last week where you have a third party internal application wanting to reference a file on a CDN. As luck would have it, that CDN has been deprecated and changing it requires rebuilding a jar where the script path is hardcoded and then repackaging the internal application that uses it. Long story short, we’ll soon be doing some yak shaving that could possibly take all day so how about we just use sub_filter instead and take a nap?

Use It

Thankfully most stock builds of nginx include ngx_http_sub_module by default. Using it is actually simple enough, just slap the following directive under your location or server directive in nginx.conf.

And that’s it! You can also use regular expression here as well. sub_filter_once indicates whether nginx should apply the replacement once or repeatedly. One gotcha you might have is that by default this only works with mime types of text/html. If you want to modify the mime-types that subfilter executes on set `sub_filter_types` to the desired type.

Give It a Try

I’ve put together a very simple demonstration of using subfilter for text replacement on Github.

Friday Functions: NPM Path

This week’s Friday function is probably the most simplistic in my zsh library. Every time I worked with node.js I always got a little annoyed with having to reference the command line apps referenced under ./node_modules/bin. Sure I could do an npm install -g but I don’t like mucking with my global environment on a per project basis. Python has virtualenvs which separate these concerns pretty well but to my knowledge node.js has no equivalent.

So my zshrc has contained this function for quite some time.

Now I just run npm path from the root of a node.js project to quickly add its bin directory to my path. I’m sure there are probably more elegant solutions these days… please let me know in the comments below! 🙂

Testing In Ansible

This is a topic I brushed up against yesterday and meant to blog about it at the end of the day but got a little busy. A lot of times when provisioning boxes locally in vagrant I’ve thought it would be incredibly useful to be able to automatically test the system to ensure all the expected bits are provisioned as expected.

I’ll probably throw together a nice public demo but the short and skinny is to include a final ansible provisioning step after the normal step that runs a test playbook of sorts against the system. For us we just dumped our test tags into our main roles and tag them as test. Then in vagrant we exclude test tagged tasks and then in the test phase we only run those tagged tasks. Below is an example for one of our services to test that two service processes are running and that the load balancer is also serving up responses that are the same as those running on the two processes.

I’ve also heard of other tools in this space like ServerSpec which may fit your bill if you’re not running ansible or are running some mixed environment. So far I think ansible fits well here but you’re definitely going to be a little limited due to the tests being in yaml. Although you could hypothetically write some custom modules or resort to shell wizardry if you need something more advanced.

I’m really excited about this… the idea we could have full test suites with each of our ansible roles that can verify a whole swath of aspects like expected ulimits and the like is GREAT.

Friday Functions: AWS ZSH Helper

This morning I’m going to go with a new recurring weekly post: Friday Functions! While some of it will aim to share my large inventory of zsh functions I’ve acquired over the years I’ll also be finding new additions if I run out of material. So it also serves to help me learn more!

This week’s function is probably only useful if you’re into AWS and use the awscli tool to interact with it from the command line. Using the awscli command direction can be quite verbose so some nice shortcuts are useful. I actually learned of this handy function from Kris’s awesome collection of zsh configuration and made a few small adaptions to it.

This is pretty useful. If you want to find all instances with http in the name you just run aws-instances-describe http.

Screen Shot 2016-03-31 at 6.14.42 PM

Or if you want to look for instances by a specific tag you can use the `-t` switch. For example, to find all instances with the worker_email role tag we can just run aws-instance-describe -t role worker_email. You can add -s to changed the filter to include the running state and like the actual call you can include multiple instances. So if you wanted to find all stopped instances with the taskhistory role you’d run aws-instance-describe -t role taskhistory -s stopped. The function sets this to default to running instances only since that’s what I’m looking for 99% of the time… looking for stopped or terminated instances is definitely the exception.

Hope this was interesting enough. Ideas, thoughts, comments or criticism are all welcome in the comments below! Let me know what you think! 🙂

Managing Dotfiles With Ansible

Yesterday I posted about managing our local configuration with Ansible and today I’m going to continue this path by putting my zsh configuration under configuration management.

Installing ZSH and oh-my-zsh

First up let’s install our preferred shell and customizations. For me this is zsh and oh-my-zsh. Up front I know that this is going to probably be a multi-step process so I’m going to create a local role to bundle up the tasks and templates we’ll be using. I create a new directory named roles and add it to the roles_path in our ansible.cfg. This new directory with our new role will look like the following.

For our initial set of tasks we’ll install zsh via homebrew, configure zsh as the current user’s shell and install oh-my-zsh.

You can see this change in its entirety here. Next up we’ll add a template for our zshrc that we can customize to our liking. To start we’ll grab the zshrc template from the oh-my-zsh git repository and save it to jamescarr.dotfiles/templates/zshrc.j2.

A good first piece of literal text to template out is the zsh theme and the plugins loaded up. We’ll define these with default variables under jamescarr.dotfiles/defaults/main.yaml.

You’ll notice here we’ll also be polite by backing up the existing zshrc file if it is present. A good benefit with this example is that we can now switch up the different pieces of our .zshrc configuration from our playbook by overriding the default variables.

You can find everything up to this point at this referenced commit in jamescarr/ansible-mac-demo.

Up Next

Obviously a topic like zsh customizations is a rather large undertaking deserving of its own post so tomorrow I’ll share some of my personal zsh functions and aliases that I find extremely useful to start off with.

Well did you find the useful? Did you run into any problems following along? Please let me know in the comments below!

Managing Your Macbook with Ansible

For a long time I’ve been a big believer in Infrastructure as Code and I have always wanted to use configuration management to provision my personal workstation and keep it constantly updated to an expected state. Boxen was one of the first tools I saw in this space and it even seemed like it might be comfortable since I was using Puppet at the time. However I never really had a lot of luck with it and the original aim of Boxen was actually lost on us at Zapier since we engineered a very nice docker-compose based setup that lets anyone begin running and hacking on zapier locally constrained by the time it takes to download the docker images for the first time.

That being said when we began transitioning from Puppet to Ansible last year and I naturally started using it locally to kind of whet my appetite a bit. Here’s a brief run down of how I’m currently using Ansible to manage my laptop and some notes on where to go next.

Getting Started

There are several guides out there on getting Ansible installed, the most authoritative being the instructions right on Ansible’s website. I won’t repeat those well written steps here.

Once that’s all done let’s run ansible --version and verify we’re running Ansible or above. If you’re visiting from the future then I will have to say that I am really unsure if this blog post will work with 3.0.0 or whatever release is out then. Keep in mind this post is written in 2016. 🙂

First up we’ll create a very simple Ansible playbook that just prints a hello world message to the console to ensure we have everything configured correctly.

Place this in a project directory (I name mine “personal-dev-laptop”) and run ansible-playbook main.yml. If all is configured correctly you’ll see a playbook run that executes a task that prints out “Hello World” to the console.


The most important piece to a provisioning system is the package management and Ansible is no different. Homebrew is the go to on OSX and thankfully Ansible has a pretty decent homebrew module for managing the state of different Homebrew packages. Let’s dip our toes in by adding a task to ensure macvim is installed and at the latest version.

The nice benefit here is that each time we run Ansible macvim will automatically get updated to the latest available package. However if we want to ensure a package is simply installed but don’t want to upgrade each time we run we can set the state to `present`. After awhile if we’ve worked with vim and decided that it’s just not for us and we’d prefer to use emacs instead we could just set macvim’s state to absent and emacs state to latest.

Taking It Further

Sure we can just keep adding our own tasks to install roles, perhaps even using a with_items iterator to include a big list of them but sooner or later we’re going to be duplicating a lot of work someone else has done. Which is a good time to introduce third party roles installed via ansible galaxy. There are most likely several good roles out there but my favorite so far is geerlinguy.homebrew. I usually put a requirements yaml file in the main root of my project with the module I want to use and the version I want to lock in.

Now to install this third party role we’ll run ansible-galaxy install -p vendor -r requirements.yaml. The -p switch will install it to a local directory named vendor so we don’t clobber the global include path and we can add that directory to our project’s .gitignore so that it isn’t stored in git. We also add an ansible.cfg to specify the role path for third party roles we’ll be using.

Now we also update our main.yaml to include a few changes. Firstly we want to include the new role we just imported and then we move the packages we want to install as variables that the homebrew role will utilize.

This time we’ll run with the -K switch since this role also ensures that homebrew is installed and will require sudo access to do so. Now I know what you’re thinking… you’re thinking “James is trying to hack my box!” and quickly closing your browser tab. Obviously you should never provide sudo without giving the source code a look over and the most important pieces will be the main task file and meta file where there could be dependent roles. After careful inspection we decide all is good and run ansible-playbook -K main.yml. Congratulations, you now have Spotify and iterm2 installed!

One small improvement to make before we move on is to extract these variables that are specifically for homebrew to their own var file. While it might seem silly now, sooner or later we might be using many roles that utilize many different variables and mixing them will lead to a lot of confusion. I personally like to name the variable files after the role they’re used for as illustrated below.

Managing OSX Settings

You can do a lot of tweaking to how your OSX behaves by using the osx_defaults module to manage OSX Defaults. There’s a lot of opportunities here but I’ll just leave a quick and dirty example to set our preferred screensaver below.

You could possibly even go as far as using this to manage various applications you have installed and possibly even setting registration keys for those applications. I haven’t even gotten to that point yet either so I’m not covering it here.

Further Reading

Well I hope this was good for you… it was good for me and helped me flesh out some of my current setup. I’m still on my path to learning how to best utilize ansible to manage my development environment so there’s definitely more to learn that I’ll continue to share as time progresses. I’m also not ignorant of a few other projects that aim to make working with ansible to manage development environments easier and one I’ve been looking at is Battleschool.

You can find the completed work for this blog post on github at jamescarr/ansible-mac-demo.

Crouching Supervisor, Hidden File Descriptor Setting

Here’s an interesting problem our team faced last month that was extremely infuriating. We were in the process of launching replacement haproxy instances that are used to load balance to nodes in our RabbitMQ cluster. We’ve done this a lot of times before and set all the usual user settings required under limits.d to ensure proper file descriptors are allocated for the haproxy process. While creating this new role we also decided to use supervisor to supervise the haproxy process as it was previously observed in an older release that it didn’t automatically restart when it crashed (which in itself is a rarity).

Everything looked solid and we began throwing some traffic at the new balancer. Eventually we discovered something had gone horribly wrong! Tons of connection refused errors began showing up and the behavior exhibited was what one would expect if file descriptors weren’t being allocated correctly. Sure enough a quick look at /proc/<pid>/limits revealed that maximum open file descriptors were set to the very low value of 1024. We directed traffic back to the old balancer and began the investigation. How could this be? All of the settings were correct so why is it being set to 1024?

Supervisor was one new variable in the mix so I decided to begin pursuing the supervisor documentation and scanning for the number 1024 to see what might be tied to that. Sure enough, I came to discover the minfds setting. Let’s take a look at what the supervisor documentation has to say about this setting.

The minimum number of file descriptors that must be available before supervisord will start successfully. A call to setrlimit will be made to attempt to raise the soft and hard limits of the supervisord process to satisfy minfds.

The hard limit may only be raised if supervisord is run as root. supervisord uses file descriptors liberally, and will enter a failure mode when one cannot be obtained from the OS, so it’s useful to be able to specify a minimum value to ensure it doesn’t run out of them during execution. This option is particularly useful on Solaris, which has a low per-process fd limit by default.

Default: 1024

Well that doesn’t make much sense… if I’m reading this correctly it’s simply saying that the number specified is the minimum that should be available, right? The devil as they say is in the details. If we look at the documentation on setrlimit we’ll clearly see that this will actually set the limits without any reservations on what it currently is. The call basically is going to set max open files to whatever the value minfds is defined to in supervisor. Sure enough, as an experiment I set minfds in supervisor’s configuration to a higher number and after restarting supervisor the number of open file descriptors allocated to the haproxy process were greatly increased and reflected what minfds was set to.

In the end this pain also turned out to be unnecessary. While we had used supervisor because it was “what we know well” it turned out that the newer distribution we were releasing on already managed services via systemd which by default was also configured to respawn on failure.


Hopefully this story will prevent a similar trail of sorrow for others who may encounter the same situation!

TLDR; If you’re having supervisor supervise an application that is sensitive to max open file descriptors you’ll want to ensure minfds is set to match!


It is interesting what you learn when you have a five year old going through kindergarten and how much the observations you make seem to reflect on life in general. While I was standing in the hallway after dropping Caitlynn off for class I was looking at the different drawings with descriptions written out by each student in her class and took notice of the interesting ones. One kid likes to curl the end of each letter that seems to be nudging towards a beginner’s journey into calligraphy. Another student uses very distinctive block letters in his writing.

In the weeks that followed I began to notice something in the work Caitlynn was bringing home. One page she used letters that exactly mimicked those written by the student who used fancy curls at the end of her letters. Another week, she used block letters and within the same week had a paper that mixed both in a way that created something truly unique. As time progressed her writing keeps developing in ways that seem to incorporate what she has seen other students do and her drawings also follow what she observes and then evolves into something truly her own.

That’s when it really struck me, something very obvious but I never took the time out to think about it. A very large portion of our behavior, knowledge and actions are learned by mimicking others. We take the knowledge and experience of others and make it our own, using it as a base to build off of. As we encounter new situations and different approaches we continue to take those experiences and add them to our own experiences. Who we chose to surround ourselves with and who we look upon as our role models heavily influence who we turn out to be.

In engineering I find this very important as a way to acquire new knowledge. We’re influenced by the people we work with or even by reading the source code of projects we admire on Github. Whether it’s different tools like tmux or certain patterns and practices we observe others using we stand back and think “Huh, that’s clever.” and find someway to incorporate it into our behavior.

It is also important to keep in mind that just as we’re influenced by the behavior of others we too are influencers. That’s why it’s important to take the experiences we learn and find a way to share them, whether it’s a blog post or a simple Github project that shares a best practice you’ve learned. You have to ask yourself if you want your discoveries to be left in the dark or available for others to learn from?

Therefore I have decided to dust off this old blog and begin blogging again because I miss sharing meaningful content. I think there can be a lot of excuses as to why I’ve dropped the ball for so long to provide useful content here but the truth is none of those reasons matter. What does matter is that I’ve decided to challenge myself this year by posting as much as possible even if it’s just a summary of some small discovery I made that day. The blog has a new lease on life again so stay tuned!