Haystack

Reviewed by Cristian
Updated May 1st, ’22

Tekpon Score

9.4

Users Score

7.5

Haystack Review

Share on

Haystack software is marketed on the official website as “building elite developer teams.” The Haystack analytics bundle consists of feedback and metrics tools and services that scour the developers’ work for patterns and anomalies. As a slogan on the website states, the goal is to detect and prevent developer burnout. The feedback loop used to assess the well-being of developers doubles as a code assessment tool for integrating the best parts of the iterative process. In short, Haystack is a team app and a workflow optimization service.

UI/UX

9.5

Ease of use

9.0

Value for Money

9.5

Features

9.5

Customer Support

9.5
Pricing Details

Starting from: $30/ month

Pricing Model: Subscription

Yes, has free trial

No free version

PROs
  • Doubles as an incident response solution
  • Metrics show the development bottlenecks
  • Minimizes friction between managers and software engineers
CONs
  • Still requires managerial skills
  • Haystack may be making a copy of the code
Best for
  • Small business
  • Medium business
  • Large enterprises
  • Freelancers
Deployment
  • Cloud, SaaS, Web-Based

What is development analytics software?

Development analytics software is also known as “Git analytics tools.” It provides insight into a software development project by integrating with the GitHub code repository, from which it draws data points.

The intended users are team leaders and managers itching for constant status updates without burdening or pressuring the programmers. The feedback informs the manager that software developers are overworked and potentially disgruntled. Managers can also exert source control using these same tools, and the wording on the Haystack website implies the support staff can access it too.

Haystack — software development metrics for engineering teams

Two entrepreneurs, Julian and Kan, founded Haystack and launched it through the 2021 YCombinator as “engineering analytics that doesn’t suck.” They dissected the most common KPI development methods and envisioned engineering tools that would help engineers and managers work better.

They decided to leverage software development metrics to deliver the three core benefits through Haystack:

  • fast experimentation
  • reliable shipping
  • burnout prevention

Fast experimentation means improving what the Haystack help documents call “Cycle Time,” the time between the first commit and the merging of a pull request, indicating development speed and team performance. Cycle Time is further broken down into development speed and review speed.

A breakdown example given is:

  • average Cycle Time = 1.40 days
  • development time = 0.96 days
  • review time = 0.44 days

Development time indicates how much time was spent on developing the new features. For Haystack programmers, it averages three days. Another tech metric is review time, which indicates whether the new features have been properly tested. For Haystack, that averages 24 hours.

Reliable shipping results from a sturdy, efficient feedback loop that includes real-time analytics and weekly reports. Managers can track and benchmark developer progress to detect and remove bottlenecks thanks to real-time data processing.

Nine bottlenecks identified by Haystack are:

  • stuck engineers
  • long compilation times
  • poor quality code
  • inefficient code review
  • poorly written tickets
  • poor communication
  • team burnout
  • context switching
  • technical debt

Burnout prevention is a set of personalized advice and actionable strategies that Haystack developers use for themselves to prevent burnout, a state of total exhaustion caused by long-term stress. It manifests as dizziness, headaches, and symptoms that resemble depression. Issues in the work environment are the foremost cause of burnout that affects at least 50% of workers in the US.

Software developers are especially prone to burnout due to four reasons:

  • unrealistic deadlines
  • lack of support and/or autonomy
  • poor communication with managers
  • self-reporting of stress levels

Early identification of burnout risk factors allows managers to address burnout before it becomes a problem that can torpedo the team’s projects. The key advantage of Haystack is the elimination of self-reporting that causes developers to downplay burnout signs and managers to respond with pats on the back until it’s too late. Haystack compiled all that advice into the “Book of Engineering Management” guide on recognizing and managing burnout.

What is Haystack software?

Haystack software results from the Haystack software engineering team’s attempt to have a more even development cycle. Haystack managers noticed a lack of activity during certain hours when looking through the daily activity graph. Talking to the developers revealed they were getting bored of monotonous work and constant status meetings. So the managers decided to introduce some fun projects for a week to remedy that.

Stand-up meetings were canceled, and developers were asked to work on a fun idea that provides value to the company. The result was the most productive week in Haystack history, with the Cycle Time improving dramatically. The managers were impressed, but that still wasn’t enough to have them dispense with stand-up meetings. So instead, they were limited to 2–3 people and spaced out.

Therefore, Haystack software is a proven method to experiment with software development and track the resulting code quality metrics. It lets teams stay on the short-term and the long-term targets while the managers aggressively improve the developer tools. Haystack calls it a “culture of continuous improvement.” According to the official website, 500 teams are already using Haystack, including Meta and Indiegogo.

How to set up Haystack?

Install Haystack’s GitHub Application with read-access in one or more of your GitHub repositories. No source code will be read or stored; the app reads metadata, such as messages, status, and tags on commits and pulls requests.

To install it, visit the Haystack GitHub App Page, click “Configure” in the top right, and select the company GitHub account to install it on. Haystack does not work on personal or inactive GitHub accounts. Choose one or more repositories to install it on and wait up to 24 hours for the Haystack API to catch on to your data. You will get an e-mail notifying you when your account is ready.

In the Haystack dashboard, create teams by clicking “Settings” and “Add team.” Then, type in the basic details and add team members using their GitHub usernames. Unfortunately, Haystack does not auto-display GitHub users who haven’t made a pull request in six months.

You can also set up filters to exclude some pull requests by filtering them according to label, title, and request type. Click “Settings” in the Haystack dashboard and click “Filters.” Type in the filter conditions (case sensitive). Finally, click “Update.”

How secure is Haystack?

According to the Haystack website’s Security Policy page, no source code data is accessed without the owner’s explicit approval and only when responding to an incident or performing support actions. This implies Haystack makes a backup copy of the source code in case a disgruntled team member decides to wreak havoc on it.

Payments are processed through Stripe, and no payment details are stored on Haystack servers. Haystack’s hosting provider is Digital Ocean; the link to their Security Policy page 404s. All traffic is encrypted using 256-bit encryption. There is no mention of Haystack undergoing an independent security audit. Haystack staff is vetted through background checks, and employees sign a confidentiality agreement.

Haystack’s security practices do not instill confidence and are unlikely to hold back a sophisticated attacker. They may work for smaller companies and teams but are not enterprise-ready. The best course of action is to assign high-risk engineers to non-essential projects and then assess their behavior through Haystack.

Haystack Features

Let’s discover some of its best features:

  • Metrics

Haystack does not provide engineering metrics in a way that would lead to toxic comparisons and a competitive atmosphere. For instance, it’s impossible to use Haystack to chart all team members on a graph or display their numbers in a table. Instead, the goal is to provide metrics that lead to team improvement and analytics development over time.

Haystack tracks the four key metrics described in the “Measuring DevOps: The Four Key Metrics” video published on the Haystack YouTube channel:

  • Cycle Time
  • Deployment Frequency (most teams deploy at least once a day)
  • Change Failure Rate (the percentage of deployments that cause issues; the average is 15%, but elite teams have it under 4%)
  • Mean Time To Recovery (time to restore service after a failure; should be automated and measured in seconds)

The video references the book called “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations,” which analyzed 23,000 data points from companies of all sizes, including startups and enterprises. Engineering teams that performed well across those four metrics provided their companies with a larger market share, customer satisfaction, and profitability.

metrics

  • Dashboard

The Haystack dashboard displays the first three metrics mentioned in the above bullet list. Instead of Mean Time To Recovery, Throughput measures the number of pull requests per day to indicate team bandwidth.

Visualization of bottlenecks through data analytics is the primary reason to use the dashboard. It should be checked at least once a day to identify which software developers might need help.

dashboard

  • Alerts

Haystack integrates with Slack to provide low-friction notifications regarding the development cycle. Described on the official website as helping “superpower your #dev channel,” Haystack sends out automated reminders, such as “PR #213 has been stuck in discussion for three days”.

Daily Health Checks help team members build healthy habits by nudging them to avoid unbalanced workloads. Daily Health Checks include the daily snapshot, summarizing what the team did and potential risks, such as merging without review. There are automated weekly recaps as well to provide more retrospective analyses.

The alerts can be customized through the metrics ranges to warn the manager that, for example, a software developer is doing too much work over the weekend. Of course, the real-time GitHub insights can be delivered through e-mail too.

alerts

  • Book of Engineering Management

The book is a 39-page PDF describing the most common unhealthy signals and patterns that indicate the team is getting stuck. Some unhealthy signals and patterns are:

  • unusually long time spent in development and/or review
  • an unusually high number of open pull requests
  • inconsistent and/or increasing Cycle Time
  • merged pull requests that go above or below the average throughput range

Some associated metrics can have a variable range, such as 70–90% for the merge rate. According to the PDF, understanding the range reveals the bottleneck and the underlying unhealthy work patterns. In the case of the merge rate, the lower it gets, the more it indicates work rolling over from one sprint to the next. The manager would have to drill down to the core issue and figure out what’s going on and why.

That reveals Haystack only works, and the manager is acting on its data. In cases where engineers are outliers or Haystack uses a small sample size, the manager might reach the wrong conclusion. The manager has to understand the context and figure out what is happening and why. Rework time is the perfect example of how data may mislead without context.

The PDF explains that rework is a part of the review process that shouldn’t take too long. If rework takes too much development time, it may indicate poor quality code or perfectionism. There’s little wrong with the code, but the engineer in question wants it perfect, which slows everyone else down. Instead of a heavy-handed approach, the manager should approach the situation gingerly and indicate that the code must be fine, not perfect.

Book of Engineering Management

Haystack Pricing

The standard pricing plan costs $20 per developer per month ($30 when billed monthly). Haystack recommends it for teams of less than 100 developers. For teams with 100+ members, Haystack has the Enterprise pricing plan, which requires calling and discussing the terms, which may not align with your GitHub Enterprise plan. There is a 14-day free trial that doesn’t require a valid payment method.

Growth

$30
Monthly
  • Unlimited Alerts
  • 6 Month Historical Data
  • Email support
  • Account Specialist Support

Enterprise

Custom Pricing
Contact them
  • Everything in Growth plus
  • On-Premise / Custom Integrations
  • Dedicated Performance Specialist
  • Quarterly Performance Reports
  • Team Training / Onboarding Sessions

See all Features

Alternatives to Haystack

Maybe this tool is not perfect for you. But, no worries, there are other alternatives that you can try.

haystack competitors

Haystack vs. GitLab

Project Haystack integrates with GitLab, which touts itself as “the one DevOps platform.” In short, GitLab is a collaboration and software deployment platform. It handles the ten crucial steps for software creation and their substeps, such as API Security and Error Tracking. There is no mention of burnout or developer well-being on the GitLab website, indicating it only cares about data. GitLab comes in three tiers:

  • Free
  • Premium
  • Ultimate

And two versions: cloud-based and local. The local version can be administered without connecting to an outside database or server but also requires you to do the maintenance on it. Not all steps are available to all pricing tiers.

Haystack vs. LinearB

LinearB is touted as a “systematic approach to engineering improvement.” It provides visibility into the engineering organization but doesn’t rely on metrics alone. Instead, the key feature is what LinearB calls “pipeline observability“. By showing how well the business goals are being fleshed out, LinearB reveals where the workflow can be improved.

The dashboard reveals a metric called “people effort,” which is designed so that CEOs can understand who is contributing and how much. There is no need to log into LinearB; it sends alerts to Slack and/or e-mail addresses. The focus is on project delivery though there are mentions of burnout prevention and cognitive overload in the help docs, with instructions on setting up the associated alerts. LinearB is completely free for individual teams.

Haystack Review Conclusions

Software engineers are notorious for being insular. They want to get the job done and move on to the next task without anyone bothering them. Managers popping in to check up on the work is a distraction that makes them retreat even more into project building. Haystack offers itself the perfect solution for checking in on software developers and reminding them of the priority task without disrupting their workflow.

Haystack connects managers to software developers in a simple, logical way, providing the former with metrics that show the latter’s progress (whether the metrics are revealed to team members is the manager’s call). Haystack is also an app that can replace stand-up meetings and discussions, redirecting time and energy into the project. Still, managers who can’t understand the context of those metrics can cause even more trouble than they usually would.

Haystack provides the most value when used to train managers to understand the code development process and contextualize the success metrics. However, it can prevent burnout only if the manager can care for the stress levels of the underlings. Besides that, Haystack should not be connected to crucial code repositories unless the company is fully aware of the risk.

Competitors

LinearB

Tekpon Score

Users Score

Updated Mar 23rd, ’22

Pluralsight Flow

Tekpon Score

Users Score

Updated Mar 23rd, ’22