Haystack Reviews

What is development analytics software?

Development analytics software is also known as “Git analytics tools.” It provides insight into a software development project by integrating with the GitHub code repository, from which it draws data points.

The intended users are team leaders and managers itching for constant status updates without burdening or pressuring the programmers. The feedback informs the manager that software developers are overworked and potentially disgruntled. Managers can also exert source control using these same tools, and the wording on the Haystack website implies the support staff can access it too.

Haystack — software development metrics for engineering teams

Two entrepreneurs, Julian and Kan, founded Haystack and launched it through the 2021 YCombinator as “engineering analytics that doesn’t suck.” They dissected the most common KPI development methods and envisioned engineering tools to help engineers and managers work better.

They decided to leverage software development metrics to deliver the three core benefits through Haystack:

Fast experimentation means improving what the Haystack help documents call “Cycle Time,” the time between the first commit and the merging of a pull request, indicating development speed and team performance. Cycle Time is further broken down into development speed and review speed.

A breakdown example given is:

Development time indicates how much time was spent on developing the new features. For Haystack programmers, it averages three days. Another tech metric is review time, which indicates whether the new features have been properly tested. For Haystack, the average is 24 hours.

Reliable shipping results from a sturdy, efficient feedback loop that includes real-time analytics and weekly reports. Thanks to real-time data processing, managers can track and benchmark developer progress to detect and remove bottlenecks.

Nine bottlenecks identified by Haystack are:

Burnout prevention is a set of personalized advice and actionable strategies that Haystack developers use for themselves to prevent burnout, a state of total exhaustion caused by long-term stress. It manifests as dizziness, headaches, and symptoms that resemble depression. Issues in the work environment are the foremost cause of burnout that affects at least 50% of workers in the US.

Software developers are especially prone to burnout due to four reasons:

Early identification of burnout risk factors allows managers to address burnout before it becomes a problem that can torpedo the team’s projects. The key advantage of Haystack is the elimination of self-reporting that causes developers to downplay burnout signs and managers to respond with pats on the back until it’s too late. Haystack compiled all that advice into the “Book of Engineering Management” guide on recognizing and managing burnout.

What is Haystack software?

Haystack software results from the Haystack software engineering team’s attempt to have a more even development cycle. Haystack managers noticed a lack of activity during certain hours when looking through the daily activity graph. Talking to the developers revealed they were bored with monotonous work and constant status meetings. So the managers decided to introduce some fun projects for a week to remedy that.

Stand-up meetings were canceled, and developers were asked to work on a fun idea that provides value to the company. The result was the most productive week in Haystack history, with the Cycle Time improving dramatically. The managers were impressed, but that still wasn’t enough to have them dispense with stand-up meetings. So instead, they were limited to 2–3 people and spaced out.

Therefore, Haystack software is a proven method to experiment with software development and track the resulting code quality metrics. It lets teams stay on the short-term and the long-term targets while the managers aggressively improve the developer tools. Haystack calls it a “culture of continuous improvement.” According to the official website, 500 teams already use Haystack, including Meta and Indiegogo.

How to set up Haystack?

Install Haystack’s GitHub Application with reading access in one or more of your GitHub repositories. No source code will be read or stored; the app reads metadata, such as messages, status, and tags on commits and pulls requests.

To install it, visit the Haystack GitHub App Page, click “Configure” in the top right, and select the company GitHub account to install it on. Haystack does not work on personal or inactive GitHub accounts. Choose one or more repositories to install it on and wait up to 24 hours for the Haystack API to catch on to your data. You will get an e-mail notifying you when your account is ready.

In the Haystack dashboard, create teams by clicking “Settings” and “Add team.” Then, type in the basic details and add team members using their GitHub usernames. Unfortunately, Haystack does not auto-display GitHub users who haven’t made a pull request in six months.

You can also set up filters to exclude some pull requests by filtering them according to label, title, and request type. Click “Settings” in the Haystack dashboard and click “Filters.” Type in the filter conditions (case sensitive). Finally, click “Update.”

How secure is Haystack?

According to the Haystack website’s Security Policy page, no source code data is accessed without the owner’s explicit approval and only when responding to an incident or performing support actions. This implies Haystack makes a backup copy of the source code in case a disgruntled team member decides to wreak havoc on it.

The company processes payments through Stripe and doesn’t store any payment details on Haystack servers. Haystack’s hosting provider is Digital Ocean; the link to their Security Policy page 404s. Furthermore, Haystack encrypts all traffic using 256-bit encryption. There is no mention of Haystack undergoing an independent security audit. Haystack staff is vetted through background checks, and employees sign a confidentiality agreement.

Haystack’s security practices do not instill confidence and are unlikely to hold back a sophisticated attacker. They may work for smaller companies and teams but are not enterprise-ready. The best course of action is to assign high-risk engineers to non-essential projects and then assess their behavior through Haystack.

Haystack Features

Let’s discover some of its best features:

Haystack does not provide engineering metrics in a way that would lead to toxic comparisons and a competitive atmosphere. For instance, it’s impossible to use Haystack to chart all team members on a graph or display their numbers in a table. Instead, the goal is to provide metrics that lead to team improvement and analytics development over time.

Haystack tracks the four key metrics described in the “Measuring DevOps: The Four Key Metrics” video published on the Haystack YouTube channel:

The video references the book “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations,” which analyzed 23,000 data points from companies of all sizes, including startups and enterprises. Engineering teams that performed well across those four metrics provided their companies with a larger market share, customer satisfaction, and profitability.

The Haystack dashboard displays the first three metrics mentioned in the above bullet list. Instead of Mean Time To Recovery, Throughput measures the number of pull requests per day to indicate team bandwidth.

Visualization of bottlenecks through data analytics is the primary reason to use the dashboard. You should check it at least once a day to identify which software developers might need help.

Haystack integrates with Slack to provide low-friction notifications regarding the development cycle. Described on the official website as helping “superpower your #dev channel,” Haystack sends out automated reminders, such as “PR #213 has been stuck in discussion for three days”.

Daily Health Checks help team members build healthy habits by nudging them to avoid unbalanced workloads. Daily Health Checks include the daily snapshot, summarizing what the team did and potential risks, such as merging without review. You also have automated weekly recaps as well to provide more retrospective analyses.

You can customize the alerts through the metrics ranges to warn the manager that, for example, a software developer is doing too much work over the weekend. Of course, the software can deliver real-time GitHub insights through e-mail too.

The book is a 39-page PDF describing the most common unhealthy signals and patterns that indicate the team is getting stuck. Some unhealthy signals and patterns are:

Some associated metrics can have a variable range, such as 70–90% for the merge rate. According to the PDF, understanding the range reveals the bottleneck and the underlying unhealthy work patterns. In the case of the merge rate, the lower it gets, the more it indicates work rolling over from one sprint to the next. The manager would have to drill down to the core issue and determine what’s going on and why.

That reveals Haystack only works, and the manager is acting on its data. In cases where engineers are outliers or Haystack uses a small sample size, the manager might reach the wrong conclusion. The manager has to understand the context and figure out what is happening and why. Rework time is the perfect example of how data may mislead without context.

The PDF explains that rework is a part of the review process that shouldn’t take too long. If rework takes too much development time, it may indicate poor quality code or perfectionism. There’s little wrong with the code, but the engineer in question wants it perfect, which slows everyone else down. Instead of a heavy-handed approach, the manager should approach the situation gingerly and indicate that the code must be fine, not perfect.

Haystack Review Conclusions

Software engineers are notorious for being insular. They want to get the job done and move on to the next task without anyone bothering them. Managers popping in to check up on the work is a distraction that makes them retreat even more into project building. Haystack offers itself the perfect solution for checking in on software developers and reminding them of the priority task without disrupting their workflow.

The development analytics software connects managers to software developers in a simple, logical way, providing the former with metrics that show the latter’s progress (whether the metrics are revealed to team members is the manager’s call). Haystack is also an app that can replace stand-up meetings and discussions, redirecting time and energy into the project. Still, managers who can’t understand the context of those metrics can cause even more trouble than they usually would.

Haystack provides the most value when used to train managers to understand the code development process and contextualize the success metrics. However, it can prevent burnout only if the manager can care for the stress levels of the underlings. Besides, Haystack should not be connected to crucial code repositories unless the company is fully aware of the risk.