16 Years of DevOpsDays: A Look at the Numbers

2026-07-31T00:00:00+00:00

Since the first DevOpsDays in Ghent in 2009, the community has organised well over 700 events across every populated continent. Here's what the shape of that history looks like.

The long arc, 2009–2026</h2>
Events grew steadily from a handful in the early years to a peak of 81 in 2020 (yes, the pandemic year). The virtual format lowered the barrier to entry: no venue costs, no travel logistics, no border controls. 2020 briefly turned every organiser's "we could totally run one" into an actual event.
2021 was the hangover. Numbers dropped nearly in half (42), reflecting both organiser burnout and the awkward middle period between virtual and in-person. From there, in-person events rebuilt: 56 in 2022, 64 in 2023, 68 in 2024.
The most recent years show a mild decline: 60 in 2025 and 53 confirmed for 2026 so far. To me, it looks less like a slump and more like a return to a stable, sustainable equilibrium.

Where DevOpsDays happen</h2>
The all-time regional picture:
Region</th> Events</th></tr></thead>

USA</td> 261</td></tr>
Europe</td> 192</td></tr>
Brazil</td> 106</td></tr>
East Asia</td> 32</td></tr>
Latin America (excl. Brazil)</td> 32</td></tr>
Canada</td> 29</td></tr>
South & Southeast Asia</td> 26</td></tr>
Africa</td> 18</td></tr>
Oceania</td> 18</td></tr>
Central Asia</td> 14</td></tr>
Middle East</td> 13</td></tr> </tbody></table>
USA and Europe together account for roughly 61% of all events ever held. That's the headline, but it undersells one thing: Brazil. 106 events makes it the single most active country after the USA (by a comfortable margin).
The post-COVID picture</h2>
Restrict the lens to 2022 and later, and the picture shifts:
Region</th> 2022–2026</th></tr></thead>

USA</td> 89</td></tr>
Europe</td> 80</td></tr>
Brazil</td> 61</td></tr>
Latin America (excl. Brazil)</td> 17</td></tr>
East Asia</td> 12</td></tr>
Central Asia</td> 10</td></tr>
South & SE Asia</td> 8</td></tr>
Canada</td> 8</td></tr>
Africa</td> 6</td></tr>
Oceania</td> 4</td></tr>
Middle East</td> 4</td></tr> </tbody></table>
A few things stand out:

Brazil is on the come-up. From 5 events in 2022 to 18 in 2025. At current rate it will outpace Europe within a year or two.</li>
The USA is slowing down. From 22 events in 2022 to 11 (so far) in 2026; a real decline, which speaks to shifting business priorities.</li>
Europe is remarkably steady. Bouncing between 14 and 18 events a year through the whole post-COVID period.</li>
Central Asia has quietly arrived. No events at all before 2018; now a consistent presence in double digits.</li> </ul>
OK, so now what?</h2>
DevOpsDays is smaller than it was at its 2019–2020 peak, but I think we're healthier for it. More distributed, less dependent on any single region, and drawing organisers in places (Baku, Almaty, Tashkent, Kazan) that would have been unthinkable a decade ago. The centre of gravity is still transatlantic, but it's drifting south and east, one event at a time.

Building a (mostly) reliable research assistant

2026-04-24T00:00:00+00:00

If you install software from a public package registry (npm, PyPI, Crates, etc.), every security decision that registry makes sits inside your trust boundary. You don't run the infrastructure, you don't set the publisher policies, you don't decide what "verified" means, and whatever the registry (fails to) enforce travels straight into your build.
This is a well-studied surface. Sonatype tracked more than 700 thousand malicious open-source packages between 2019 and the end of 2024, with over half a million new ones in the last year of that window alone! That's a 156 percent year-over-year growth rate.^{1</a> There are some existing concepts that can help: SLSA gives us a vocabulary for provenance2</a>, for example, and Sigstore gives us signing and transparency logs.3</a> Both are real, serious, and widely adopted. But provenance and signing don't themselves answer the question I keep bumping into, which is what is the actual security posture of package registries?}
So I decided to try to answer that question. I started where all good plans start: a list. I call it the Software Supply Chain Security Scorecard (SSCSS). It's a set of 52 binary questions across eight sections, each probing one concrete security control a registry either enforces or doesn't: Does the source enforce multi-factor authentication for publishers? Does it require identity federation for organisational accounts? Does it display a verified publisher identity to consumers? Does it have a published vulnerability disclosure policy? Yes, no, or unknown. The questions are binary, but the work of answering them is anything but.
A quick note: This is not to be confused with the OpenSSF Security Scorecard project4</a>. This is an unrelated tool that scores the security hygiene of open-source source-code repositories (branch protection, signed releases, SAST configuration, and so on) rather than registries. Similar name but a different scope. (I didn't actually know it existed when I named mine.)
A good human assessment (i.e. me) takes upwards of a day and a half. It takes that long because enforced is not the same as supported; because documentation and behaviour don't always agree; because the useful evidence is usually scattered across docs, RFC repositories, governance pages, changelogs, and the occasional blog post from 2022; and because the honest answer to a surprising number of questions is still "yeah, I dunno" after hours of careful reading. Multiply that by the long tail of registries in general, and it rapidly becomes clear that if you want to cover this ecosystem with any breadth you need either a bigger team or a different approach.
Well I'm just one person, so I tried the different approach. I built an AI research agent that could work through the scorecard on its own. So this post is about what that took, which turned out to be much more than "wire up an LLM and let it go." The title's hedge is load-bearing: the agent I ended up with is mostly reliable, and the mostly is where the interesting lessons live (I think).
p.s. When I say that I built an agent, that makes it seems like it was a success from day one. It really was not. I'll walk through four attempts, and spoiler alert: the first three were failures.
Attempt one: straight at the wall</h2>
The first version was made before I had thought carefully about anything. I fired up Claude Code and gave it a single prompt that said, roughly, here is a list of 52 questions and the name of a target registry, then I let it rip. You can imagine how that went. Context limits, rate limits, truncated output, the whole thing keeling over somewhere around question fifteen. I sat with the broken output for a while and felt stupid.
In retrospect, this was the best possible outcome. It failed immediately, unambiguously, and cheaply. No half-working demo to mistake for progress, no partial results to rationalise. It was the kind of failure that makes you stop typing and start thinking. The thing I was thinking about, initially, was the shape of the input: maybe the prompt was too long, maybe the questions were too varied, maybe I needed to batch. What I should have been thinking about (and wasn't yet) was the shape of the problem. I'd get there eventually, but I had to go the long way around first.
Attempt two: the orchestration era</h2>
The second version was much more considered and approximately just as useless. Its architecture was the one you'd draw on a whiteboard if someone gave you the task for the first time: a lightweight orchestrator on Haiku, spawning a Sonnet researcher agent per scorecard section, each writing its results back to a shared JSON file. Eight parallel researchers, a coordinator, and a shared state file. Very reasonable and, as it turned out, very error-prone. The problems came in two waves.
The first wave was mechanical. Researcher agents would occasionally hang indefinitely, producing nothing and blocking the run. Others would hit a transient error mid-research, write a blank result, and report success upstream to the orchestrator, which would dutifully record the blank result as an answered question. The shared-state file became a source of subtle corruption as different researchers arrived at slightly different interpretations of its structure under slightly different success and failure conditions. The orchestrator grew more defensive with every iteration. Its prompt climbed from a few hundred lines of markdown to nearly a thousand as I added more rules for validating results, more recovery paths for hanging agents, more invariants to protect the state file. More instructions did not produce more reliable behaviour—they produced more surface area for things to go sideways on. The system became harder to reason about with every attempted fix.
I should have paused and asked why a thousand-line prompt was making things worse. I did not. I kept going. Eventually the mechanical problems settled down to a grim steady state where the system would complete a run without actually crashing, and I was able to sit down and read the results.
Which is when the second wave hit: The actual research wasn't any good.
I had confident "yes" answers backed by evidence that clearly said "no" if you read it. I had notes that bore no relation to the question they were attached to. I had URLs that did not exist. Bafflingly, I had answers where the "evidence" turned out, on inspection, to be the model narrating its own search process. Amazing. And when I ran the same assessment against the same registry three times, the results disagreed with each other—not only in their answers but in which questions they considered answerable at all. The inconsistencies were, themselves, inconsistent.
There is a name for this category of failure in the academic literature, and it is not hallucination, at least not in the useful sense of that word. Farquhar, Kossen, Kuhn, and Gal published a paper in Nature in 2024 that carves out a specific subtype they call confabulation: "arbitrary and incorrect generations" that vary between reruns of the same query.5</a> Hallucination is the umbrella; confabulation is the subset where the wrongness isn't stable. The distinction matters, because confabulations are exactly the errors you'll catch if you run a query multiple times and compare (and exactly the ones you'll miss if you don't).
Why does this happen at all? An OpenAI paper from September 2025 makes a mechanistic argument I find persuasive: modern LLM training and evaluation procedures reward guessing over acknowledging uncertainty. "Language models are optimized to be good test-takers," write Kalai, Nachum, Vempala, and Zhang, "and guessing when uncertain improves test performance."6</a> The structural incentive of the training pipeline is to produce a confident answer, not to produce no answer when no answer is warranted. Confident wrongness isn't a bug; it's the predictable output of the optimisation you've run.
You can zoom out further. In the bluntly-titled 2024 paper ChatGPT is bullshit, Hicks, Humphries, and Slater argue that "hallucination" is actually the wrong metaphor for what LLMs do when they produce false text. Hallucination implies a perceptual system misperceiving reality. But LLMs don't perceive reality—they produce text without any epistemic stake in it at all. Their behaviour is better described, the authors argue, in the sense that Harry Frankfurt gave the word bullshit in 1986: "speech or text produced without concern for its truth." The bullshitter "does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose."7</a>8</a>
This was the frame I needed. The agent wasn't misperceiving the documentation. It wasn't trying to mislead, either. It was a confident pattern-completer optimised to produce a plausible answer, and plausible answers are easier to generate than correct ones. Once I really internalised that, I stopped trying to make the system more sophisticated and started trying to make it more accountable.
I didn't get there yet, though. First I needed to take a detour through the land of milk and honey, also known as production-grade infrastructure.
Attempt three: production-grade reliability theatre</h2>
I was struggling enough with attempt two that I decided to change tools entirely. I switched to Codex and spent a couple of weeks building what I thought was going to be the serious version.
What I produced was several thousand lines of best-practice Python: a clean orchestration layer, JSON schema enforcement on every boundary, a test suite with fixtures for each question type, local MCP servers, and a sandboxed research environment with controlled network egress. It was the kind of codebase you would be quietly pleased to ship. It had docstrings. It had a linter config. The CI ran on every commit.
It made no difference.
Every time I encountered a new failure mode, more code got written to handle it. The infrastructure became genuinely robust! I could catch a hung researcher, retry a malformed search, validate a schema, roll back a corrupted state file. That was all lovely and made my nerd brain hum, but none of that touched the thing that was actually broken. The research quality didn't move. The model still confabulated. The confident-yes-backed-by-evidence-that-says-no problem was exactly as prevalent as it had been in attempt two, now wrapped in a prettier wrapper.
This is the central engineering lesson I took from the project, and I'd like to state it as plainly as I can: You cannot code your way out of an LLM quality problem.
Mechanical failures and quality failures look similar from the outside. Both produce wrong answers. Both break user-visible behaviour. Both make you feel like you need to fix something. But they have entirely different causes.
A mechanical failure is a hung process, a schema violation, a corrupted file, a network timeout, etc. These are the kinds of things our field has spent fifty years learning to handle, and for which the standard defensive-engineering toolkit works well. A quality failure is the model confidently producing content that is wrong. No amount of retry logic, schema validation, sandboxing, or test coverage touches that, because none of those tools are in the loop where the wrongness actually gets introduced. They're upstream and downstream of it, not inside it.
This is not a new observation. Chip Huyen made it in its more general ML form back in 2022: "ML systems often fail silently … Users encounter painfully wrong outputs without realizing they're wrong, making performance expectation violations harder to detect than operational failures."9</a> Hamel Husain and Shreya Shankar (who between them have written much of the current practitioner canon on evaluating LLM applications) make a sharper version of the point: the most important activity in evals is error analysis, not infrastructure. "Start with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes."10</a> Eugene Yan says it a third way: "While building with AI can feel like magic, building AI products still takes elbow grease. If teams don't apply the scientific method, practice eval-driven development, and monitor the system's output, buying or building yet another evaluation tool won't save the product."11</a>
This was the pivot. The problem was never the infrastructure. It was the method.
Factory vs lever</h3>
When I came back to Claude Code from Codex, I noticed something I hadn't seen clearly before. Both are terminal REPLs over an LLM. Both do broadly similar things. But they have fundamentally different defaults.
Codex reaches for infrastructure. Given a problem, it wants to build a factory: structured code, generator modules, layered abstractions, a test harness. It is very good at that, and if the problem you have is "produce a well-engineered software artefact," the factory instinct is the right one.
Claude Code reaches for the simplest mechanism that works. Given the same problem, it gravitates toward a single-file script, a direct invocation, the least amount of apparatus that can plausibly do the job. It will build infrastructure if you ask it to, but it won't go looking for excuses to.
This isn't a claim about the underlying models — Codex and Claude Code both run various members of the same family of frontier LLMs. It's a claim about the harness. The HumanLayer team have an apt framing: "The most well-known examples are Anthropic's Claude, where the same Claude model powers different interfaces with radically different behavior due to the harness."12</a> What you think of as a model's personality is very often a harness's personality.
For a problem where the hard part was infrastructure, Codex's defaults would have been the right ones. For a problem where the hard part was research quality, I needed a lever, not a factory. I had been fighting a tool's instinct rather than matching it. Switching tools didn't solve my problem — it just moved me from a harness whose defaults were working against me to one whose defaults were working with me.
Attempt four: starting from constraints, not architecture</h2>
For the fourth attempt I did something I should have done at the start and hadn't: I stopped designing and wrote down the constraints.
Three pages of them. Not requirements in the usual sense — no "the system shall" language. Just a list of problems I had hit and things I needed any real solution to satisfy. Answers must be grounded in verifiable quotes, not in inference. The same question run twice should produce the same answer, or at least tell me honestly that it doesn't know. The agent should be able to resume if my laptop goes to sleep. I should be able to read the agent's reasoning trail after the fact, in full. And so on.
One constraint, once I wrote it down, changed everything: Time doesn't matter.
I was assessing six registries, with more on the roadmap. I was not in a hurry. A single agent that finished a run in forty minutes of patient, serial work was strictly better than a fleet of eight parallel agents that finished in four but disagreed with themselves half of the time.
This eliminated parallelism. Which eliminated the orchestrator. Which eliminated the shared-state file. Which eliminated most of the surface area where attempt two had gone wrong. I realised that a single agent, working slowly and correctly, through all 52 questions in series. was not a compromise; it was the right answer to the actual problem, which I had been talking myself out of by building from the start.
The design principle that crystallised out of this (and which is a Datadog maxim): simple, but not simplistic. Not a short prompt that glosses over the hard parts. Not a naïve architecture that ignores the failure modes. Just the minimum complexity the problem actually demands, applied precisely where it's needed.
Anthropic's engineering team published a post called Building effective agents in December 2024 that captures the same sentiment from the other direction. Schluntz and Zhang write: "The most successful implementations weren't using complex frameworks or specialized libraries. Instead, they were building with simple, composable patterns... Success in the LLM space isn't about building the most sophisticated system. It's about building the right system for your needs."13</a> Simon Willison, after two years of refusing to use the word "agent" because nobody could agree on a definition, recently arrived at one that is about as tight as I've seen: "An LLM agent runs tools in a loop to achieve a goal."14</a> That's the whole abstraction. Everything else is implementation detail.
The system I ended up with has two skill/agent pairs. In Claude Code terminology, a skill is a named entry point (i.e. something you invoke with /skill-name</code>) that loads context and sets a workflow in motion. An agent is a specialised subprocess with its own system prompt, model assignment, and tool permissions. Skills are the user-facing verb and agents are the thing doing the work.
/build-starter-kit <registry> → starter-kit-builder (Sonnet) ↓ internal/starter-kits/<registry>.json ↓ /assess <registry> → assessment (Opus) ↓ assessments/<registry>/runtime-<ts>.json assessments/<registry>/scorecard-<ts>.json</code></pre>
Start at the start</h3>
/build-starter-kit</code> runs first, once per registry. Its entire job is source configuration: where should the assessment agent look? It maps out the registry's documentation structure, identifies the handful of hub URLs that matter (documentation root, blog root, security page, policy index), locates the RFC or proposal system if one exists, determines which URL patterns to block (package pages, user profiles, search results — anything that reflects user-generated content rather than registry policy), and captures a short interview's worth of human-only knowledge: any questions you already know the answer to from private sources, any questions known to be permanently unanswerable. It writes a starter-kit JSON and stops.
The brief is explicit about what it is not: the starter-kit builder does not read policy content, does not try to answer scorecard questions, does not populate anything from its own research. If it finds itself going deep into the registry's documentation to determine whether a security control exists, it has gone out of scope. The prompt says this three separate times in three slightly different ways because I found that the model, given the option, would drift toward helpful research every single time.
I run the starter kit on Sonnet. This is because Sonnet is extremely good at driving at the most obvious interpretation of a task by the most direct path; that tendency is a liability for the open-ended research job, but it is exactly the right instinct for source configuration, where the most obvious hub URLs usually are the right ones and you don't want the model to get clever.
Here's what a sample starter kit looks like, trimmed for length:
{ "schema_version": "1.0", "registry": "rubygems", "registry_url": "https://rubygems.org", "notes": "Operated by Ruby Central (nonprofit). Completed first external security audit by Trail of Bits in 2024. Trusted Publishing via OIDC launched December 2023. Mandatory MFA enforced for top-100 gem owners...", "sources": { "primary": [ { "url": "https://rubygems.org/pages/security", "description": "Security disclosure policy and HackerOne details" }, { "url": "https://rubygems.org/policies", "description": "Policies hub: acceptable use, copyright, privacy, ToS" }, { "url": "https://guides.rubygems.org", "description": "Documentation hub: security, trusted publishing, MFA" }, { "url": "https://blog.rubygems.org", "description": "Official blog: releases, security updates, features" } ], "secondary": [ { "url": "https://github.com/rubygems/rubygems.org", "description": "Registry source code, issues, PRs" } ], "blocked": [ { "pattern": "rubygems.org/gems/*", "reason": "Individual gem pages are user content, not policy" }, { "pattern": "rubygems.org/profiles/*","reason": "User profile pages are user content, not policy" } ] }, "rfcs": { "url": "https://github.com/rubygems/rfcs", "acceptance_criteria": "RFC PR merged into the text/ directory. Merged RFCs become 'active' after a 7-day FCP.", "notes": "Concrete examples: text/0007-mfa-rollout.md (MFA rollout), text/0010-OIDC.md (trusted publishing)." }, "pre_answered": {}, "evidence_gaps": {} }</code></pre> Nothing exotic. A list of hubs, a list of blocked patterns, an RFC pointer with a concrete definition of what "accepted" means, and room for human-only knowledge. It takes about ten minutes to produce and it is the single most important piece of context the assessment agent will ever see.
Do the actual thing</h3>
/assess</code> is where the real work happens. The assessment agent works through all 52 questions in section order. For each question it follows a source hierarchy: primary sources first (the hubs the starter kit identified), then RFCs if applicable, then secondary sources, and only as a last resort the open web. It is not permitted to answer a question from inference; every answer must be accompanied by a direct quote from a real URL. It produces two files: a scorecard-*.json</code> with the final answers and their citations, and a runtime-*.json</code> with the complete evidence trail — every query issued, every URL visited, every candidate quote, every decision point. Same timestamp for both. The runtime file is resumable: if the run is interrupted, I can pick it up on the same question where it stopped.
The assessment agent runs on Opus. Opus has a specific trait that Sonnet does not, which is that it likes to play with language. Left to its own devices, it will re-frame a question, re-phrase a search query, try a synonym, worry at a word. That is a liability when you need a direct path to an obvious answer. It is a feature when you need to interrogate a specific claim from different angles before you trust it. I'll come back to that.
One piece of context that matters before the next section. Earlier I said that running the same assessment three times produced three different answers. That wasn't a quirk of the attempt-two architecture; it's a property of LLM inference in general, and it's worth understanding why. Horace He and colleagues at Thinking Machines Lab published an essay in September 2025 called Defeating Nondeterminism in LLM Inference that pins the blame on a specific cause: "The primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies."15</a> The floating-point non-associativity story you hear in passing (different GPUs, different rounding) is the symptom; the underlying cause is that production inference systems batch requests together and the batch composition varies. Which means — and this is the useful bit — run-to-run variance is a systems-design property, not a property of the universe. It can be reduced, designed around, or embraced. You just have to pick.
Thus, my design does not try to eliminate variance. It logs it exhaustively and exploits it.
Getting the answers right: operational versus behavioural prompting</h2> The assessment agent's system prompt is about two hundred and fifty lines of markdown. Every line is load-bearing because every line addresses a specific failure I observed in a previous run's evidence trail. The prompt has grown, but grown in a different sense from the attempt-two orchestrator: not accumulating defensive handlers, but encoding lessons. The prompt splits cleanly into two kinds of instructions, and the distinction between them is the single thing I most wish I'd understood much earlier in my journey of discovery: operational vs behavioural instructions. Operational instructions tell the agent what to do and in what order, using which tools. They are procedural: parse these arguments; load the starter kit file; initialise the runtime file with this schema; call jq</code> to extract questions matching the filter; write the runtime file after each completed section, etc. Operational instructions are the kind of thing you'd write for a shell script if a shell script could handle the task, and indeed wherever a piece of operational behaviour is genuinely deterministic the right move is to pull it out into an actual shell script and have the agent invoke that. A specific example is "Use jq</code> to manipulate JSON. Do not parse or edit JSON as text." because the model, left to its own devices, will happily reimplement JSON parsing and get it subtly wrong. This generalises into a principle I've ended up putting at the top of most agent prompts I write now: tooling over LLM. The model should be the orchestrator and the reasoner of last resort. For deterministic sub-tasks—parsing, filtering, counting, scheduling, any operation where we have fifty years of reliable tooling already—use that tool. Don't let the model confabulate its way through work that a deterministic two-line jq</code> expression would get right every time. Behavioural instructions tell the agent how to think about the problem. They are about judgement: what counts as evidence, when to keep searching, how to interpret ambiguous documentation, what "enforce" means in contrast to "support." They cannot be replaced with a shell script, because the work they describe is exactly the work we are hiring the model for. You can see the split most clearly in how I write new rules. Every time I review a runtime file and find a failure mode, I ask: is this a thing the model got wrong because I told it to do the wrong procedure, or because it reasoned in a way I don't want? Procedural failures get operational rules: "When you cite evidence, record url</code>, quote</code>, and source_type</code> in a single object, not as parallel lists." Reasoning failures get behavioural rules: "A quote must be content from the source, not a description of your own search process." Confusing the two produces prompts that are both longer and less effective. A small team of researchers with varying expertise</h3> I find it useful to think about agent prompts the way I'd think about briefing a team of contractors. You would not walk into a room of six analysts and deliver one monolithic paragraph of instructions. You would break the work into specified sub-tasks with clear handoffs; you would attach behavioural norms to each one: "be skeptical of vendor marketing," "quote sources verbatim," "keep searching when the evidence is ambiguous", and so forth. Naturally, each member of the team has varying expertise, which means things that seem obvious might need to be spelled out (sometimes in excruciating detail). The agent prompt is therefore that briefing document, written for an extremely fast but extremely literal contractor who has never worked with you before and forgets everything between engagements. Things like, "please don't make things up," "don't stop at the first promising result," "here is what I mean by 'enforce'". You have to say these things to the agent, because it will assume whatever you don't specify, and those assumptions are probably wrong. The prompt length is important. If it's too long the model will get confused; but if it's too short, the assumptions creep in. Unfortunately there is no One True Way to get this right—you just have to tune it as you go. Non-determinism can be your friend</h3> The single most interesting thing in the behavioural half of the prompt is the way it treats non-determinism. I had spent attempts two and three trying to suppress the model's variance, and it never worked. In attempt four, I flipped the polarity. Every question gets re-phrased in three different ways before any searching happens at all. The instruction, verbatim from the prompt: **Step 1 — Generate queries**: produce 3 query variations from the question text covering different angles: - Direct phrasing of the question - Synonym-heavy rephrasing - A different aspect or implication of the question</code></pre> Three variations, three different search passes per source, all three run even when the first one returns a promising result. This is weaponised non-determinism. The model's tendency to riff on language—the very thing that made Opus annoying for the starter-kit job—becomes the mechanism by which we can interrogate a claim from angles that I wouldn't have thought of. If three different phrasings of "does this registry enforce MFA for publishers?" all return the same answer from the same source, I can believe the answer. If they don't, I have contradictory evidence to work through, which is fine. Contradictory evidence is information. Missing evidence is not. The underlying pattern has academic support. Gao and colleagues published a paper in 2022 called Precise Zero-Shot Dense Retrieval without Relevance Labels that introduces a technique called HyDE — the model generates hypothetical answer documents for a query, and those documents are then used as the retrieval key.16</a> It's a particular application of a general insight: one phrasing of a question is a sample, not a signal. Sample more. And running all three variations even after a promising first result is a direct hedge against confirmation bias, which is a real and present threat with these models. The prompt is explicit about it: Run ALL 3 even after a promising result on query 1 — seek contradictions, not just confirmation.</code></pre> This is one of the patterns of what I think of as a rule ↔ failure-mode table: a handful of behavioural lines, each of which exists because the model, unsupervised, failed in that specific way often enough that I had to write the rule to make it stop. Rule</th> Failure mode it addresses</th></tr></thead> "Inference is not evidence. Every answer requires a direct quote from a source URL."</td> Model confabulates plausible-sounding justifications in place of sourcing.</td></tr> "Run all three query variations even after a promising first result."</td> Confirmation bias — model stops searching the moment it finds a confirming sentence.</td></tr> "A quote must be content from the source, not a description of your own search process."</td> Model narrates its own activity and treats the narration as evidence.</td></tr> "Ambiguous evidence means keep searching, not stop."</td> Default inclination toward closure — model wants to finish a question, not leave it open.</td></tr> "Corroborating evidence from a different URL is desirable; from the same URL it is noise."</td> Model double-cites the same page to appear to have more evidence than it does.</td></tr> "When a page contains both historical/transitional and current-state language, always prefer the current-state sentence."</td> Model quotes an older announcement while ignoring the newer one on the same page.</td></tr> </tbody></table> Every one of these rules took at least one painful runtime-file review to discover, and most of them I got wrong the first time. "A quote must be content from the source, not a description of your own search process" in particular was a bizarre pathology to encounter: the model would, when it couldn't find evidence, write a sentence describing its own search and put that in the evidence field. Things like: "Review of the security documentation did not reveal an enforcement policy for MFA." That is not evidence, but it looks like evidence to the model, which is much worse. There's a deep result here that Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, and Arawjo named in a UIST 2024 paper called Who Validates the Validators?: they call it criteria drift. Users building LLM applications, they observed, "need criteria to grade outputs, but grading outputs helps users define criteria."^{17</a> You cannot write all of your evaluation rules in advance. You have to look at outputs, notice where they go wrong, and write a rule. Then you look at more outputs, notice that a rule you wrote has a loophole, and refine it. The rule ↔ failure-mode table above didn't come from thinking carefully about the problem. It came from reading hundreds of runtime files and writing down what I saw.} This is a big part of what observability looks like for an LLM agent. Not a dashboard of request latencies (though that can be interesting, of course). Instead, what was useful to me was a full transcript of every decision the agent made, in a schema I can read, for questions I know the answer to, so I can compare what the agent did to what it should have done. Every entry in the runtime file is a sample in the distribution of the agent's behaviour, and the prompt is the mechanism I have to shape that distribution. What it actually looks like</h3> Here's a slice of a real runtime file: the agent's work on Q1.1 ("Does the source enforce multi-factor authentication for publishers?") from a RubyGems assessment. "Q1.1": { "question": "Does the source enforce multi-factor authentication (MFA) for publishers?", "status": "answered", "answer": "NO", "searches": [ { "query": "site:guides.rubygems.org MFA enforce multi-factor authentication required publishers", "url_scope": "guides.rubygems.org", "result_url": "https://guides.rubygems.org/mfa-requirement-opt-in/" }, { "query": "site:blog.rubygems.org MFA enforce mandatory multi-factor authentication all users", "url_scope": "blog.rubygems.org", "result_url": "https://blog.rubygems.org/2022/08/15/requiring-mfa-on-popular-gems.html" }, { "query": "site:blog.rubygems.org mandatory MFA all gem owners require 2024 2025 2026", "url_scope": "blog.rubygems.org", "result_url": "https://blog.rubygems.org/2023/12/14/trusted-publishing.html" }, { "query": "rubygems.org mandatory MFA all users 2025 2026 enforce requirement", "url_scope": null, "result_url": "https://rubycentral.org/news/securing-rubys-future..." }, { "query": "rubygems.org MFA enforcement expanded lowered threshold 2024 2025 all gem owners", "url_scope": null, "result_url": null }, { "query": "site:github.com/rubygems/rfcs MFA rollout mandatory all users", "url_scope": "github.com/rubygems/rfcs", "result_url": "https://github.com/rubygems/rfcs/blob/master/text/0007-mfa-rollout.md" } ], "evidence": [ { "url": "https://blog.rubygems.org/2022/08/15/requiring-mfa-on-popular-gems.html", "quote": "Today (August 15th, 2022), we will begin to enforce MFA on owners of gems with over 180 million total downloads.", "source_type": "primary" }, { "url": "https://guides.rubygems.org/mfa-requirement-opt-in/", "quote": "You can opt-in a gem you are managing by releasing a version that has metadata.rubygems_mfa_required set to true.", "source_type": "primary" }, { "url": "https://github.com/rubygems/rfcs/blob/master/text/0007-mfa-rollout.md", "quote": "Once these policy changes are fully complete for maintainers of the most popular gems, we intend to increase coverage by extending the MFA requirement to more gems in future.", "source_type": "rfc" } ], "notes": "RubyGems enforces MFA only for owners of gems exceeding 180 million cumulative downloads (roughly the top 100 gems), not for all publishers. For other gems, MFA enforcement is opt-in via the rubygems_mfa_required metadata flag set by individual gem maintainers. The accepted RFC (0007-mfa-rollout) describes phased expansion but defers broader enforcement to a future RFC that has not materialised. Ruby Central encourages MFA adoption but has not mandated it universally." }</code></pre> Six searches across three domain-scoped sources, two open-web fallbacks, and an RFC. Three evidence items from three different URLs, each with a direct quote and a source-type tag. A notes field that a human reviewer can read and understand the answer from. The full trail is much more informative than a yes/no bit on its own would be, because the reasoning trail is right there. Trust, but verify</h3> One behaviour this example demonstrates that I want to call out specifically is the theory-vs-reality gap. This one is informal security folk knowledge and, as far as I can tell, isn't systematised anywhere in the literature. The security community has rigorous frameworks for lots of things, but the gap between what a registry (or anything, really) documents about itself and what it actually does remains a largely informal concern. The agent surfaces it every few registries. Q1.1 is a soft example: RubyGems documents a phased MFA policy that sounds mandatory at a quick read but turns out, on careful reading, to apply only to the top 100 gems by download count. Other questions have turned up harder examples. Treating documentation as a starting point for investigation rather than ground truth, and forcing the agent to quote the enforcement language itself rather than paraphrasing its flavour, is one of the main things that lifts the agent above a glorified documentation summariser. There's a closely related failure mode worth calling out because it's specific to this kind of research and because an LLM handles it badly by default: absence of evidence. If the agent searches the documentation for "vulnerability disclosure policy" and finds nothing, a human researcher reads that as meaningful evidence that the policy probably doesn't exist. The agent reads it as evidence that it hasn't found the answer yet. It keeps searching, dutifully, and will eventually reach for the open web or mark the question unknown rather than drawing the conclusion a human would draw in thirty seconds. Altman and Bland's famous note on this distinction — Absence of evidence is not evidence of absence — was written about clinical trials in the BMJ in 1995, but it applies here, too: a null result is not the same as a negative result, and how you handle the distinction is a design choice.18</a> I resolve it by instructing the agent to exhaust its sources before marking a question unknown</code>, and by treating unknown</code> as a first-class answer—not a failure mode, not something to avoid, but an honest reflection of what the public documentation does and doesn't tell us. The human in the loop</h2> Even with all of the above, the agent gets some questions wrong. This is expected, because confabulation is a property of the underlying system, not a bug I can fix in a prompt. The question is what you do with the wrongness. What I do is run each assessment multiple times. Typically three, sometimes more if I have the patience. Each run is cheap: I'm not in a hurry, the agent works in series, I can do something else while it runs. I do this because the runs tend to disagree with each other in a very specific way. Most questions stabilise. Let's say that 90% of them answer the same way every time, with the same evidence, and when I spot-check those against the documentation they're right. A small group (typically three to five questions, depending on the target) flip between runs. Sometimes between YES and NO, sometimes between one of those and UNKNOWN, sometimes cycling through all three. Those flips are precisely what Farquhar et al. call confabulations: rerun-variant arbitrary wrong answers. The flip rate is, in effect, a probabilistic quality signal that doesn't require a ground-truth oracle. I don't need an answer key. I only need to notice that the model hasn't converged on an answer across multiple tries, and that by itself tells me the question is above the agent's reliable resolution. Those flipping questions are exactly where I spend my review effort. I go look at them myself, read the documentation carefully, follow the evidence the agent did and didn't find, and produce an answer I'm willing to stand behind. Then I put that answer into the starter kit's pre_answered</code> block, with the URL and the quote I used to resolve it. The next run (and every run after it) starts by copying that pre-answered entry into the scorecard verbatim and moving on. This is the part I like best about the design. Every assessment makes the next one faster and more accurate. The system accumulates a data asset over time; the starter kit starts out containing hubs and blocked patterns, and over the course of a few assessments, it gains a growing set of human-resolved answers to the specific questions the agent couldn't handle. My human effort isn't thrown away on each run: it's captured in a durable input to the next one. I hesitate to call this a ratchet because ratchet implies irreversibility, and human-resolved answers can be wrong or go stale. The starter kit's pre-answered section is versioned with the registry's policy as of a given date, and we'll need to revisit it when things change. But within a normal cadence of review, the direction is one-way: the work the agent can do reliably, it does, at scale; the work it can't, I do, once, and capture the result. Breadth from the agent, depth from me, and both compounding into a system that covers more surface area with every registry we touch. The agent itself is an attack surface</h2> I wasn't originally going to mention this, but it's important—especially for an audience that thinks about security for a living. Automated agents that have access to the open web are dangerous. An LLM agent that browses the web (i.e. reads arbitrary pages, follows links, extracts text it then feeds back into its own reasoning) sits inside a threat model that is fundamentally different from one that only operates on trusted local inputs. Simon Willison coined a phrase for it in June 2025 that I've found clarifying: the lethal trifecta. Three capabilities that, when present together, make an agent a serious security liability: Access to your private data</li> Exposure to untrusted content</li> The ability to externally communicate in a way that could be used to steal your data19</a></li> </ol> A registry-assessment agent has all three. It has private data: my local environment, whatever else is on my machine that Claude Code can see. It has exposure to untrusted content: every web page it fetches is attacker-controlled from the agent's perspective. And it obviously has external communication: it can make outbound HTTP requests. The specific threat this creates is indirect prompt injection. A hostile web page can contain instructions aimed at the agent rather than the human reader. Stuff hidden in HTML comments, like in text with CSS that sets it to display: none</code>, in image alt text, or in JavaScript that modifies the rendered content. Those instructions get read by the agent as part of its retrieved context. Greshake and colleagues formalised this class of attack in a 2023 paper called Not what you've signed up for: "LLM-Integrated Applications blur the line between data and instructions... Adversaries can remotely exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved."20</a> It is not a hypothetical concern. Brave Software's security researchers published a technical write-up in August 2025 of an indirect-prompt-injection attack against Perplexity's Comet agentic browser. The money line: "When an AI assistant follows malicious instructions from untrusted webpage content, traditional protections such as same-origin policy (SOP) or cross-origin resource sharing (CORS) are all effectively useless."21</a> The browser-level security model we've spent thirty years building does not apply at the level where an AI agent is making decisions, because the decisions are happening in LLM-land, not in browser-land. My response to this is to run Claude Code, and therefore the assessment agent, inside a sandbox called Nono. Nono is a capability-based sandbox that uses kernel-level isolation (Landlock on Linux, Seatbelt (sandbox-exec</code>) on macOS) to restrict a process to an explicit allow-list of filesystem paths and network endpoints.22</a> Once a profile is applied, the restrictions are irreversible and inherited by every child process. There's no API to loosen them from inside, including for Nono itself. If the agent is instructed by a hostile page to exfiltrate my SSH keys, it can try, but the attempt is blocked at the kernel level before any connection is made. Nono is maintained by Luke Hinds, who also created Sigstore, so the provenance on the sandbox itself is reassuring. I don't mention Nono as a product pitch. I mention it because the threat model is real, it is named, and I would rather run a browsing agent inside a kernel-enforced capability sandbox than trust the LLM to follow my instructions under adversarial conditions. The lethal trifecta is a concrete list of capabilities; if your agent has all three, it is worth taking a beat to think about what happens when the untrusted content in the middle position starts fighting back. Anyway, back to the regularly scheduled programming. OK, but does it work?</h2> I ran the same 52-question set through Gemini Deep Research as a sanity check before I started work on the fourth attempt. Gemini Deep Research is a reasonable state-of-the-art benchmark for AI-assisted research (it's the product that Google points you at if you ask for "do some research on topic X and produce a report.") I fed it the full 52 questions and the same registry context I was giving my own system, and compared the outputs. The results were instructive. Gemini hallucinated questions that weren't on the scorecard. It produced sections that were off-topic for the questions it did answer. Some answers were wrong in ways that were hard to miss; others were wrong in ways I only noticed because I happened to know the answer. And it disagreed with itself on reruns. Its run-to-run drift was roughly 20 percent, so about one question in five would change its answer between runs, or not get answered at all. This gave me a concrete target, and my 7 to 10% drift was a clear improvement on that benchmark. But the drift rate isn't really the headline. The headline is that I know which questions drift. The four-to-five-questions-per-run figure isn't a diffuse uncertainty spread over the whole scorecard; it's a concentrated uncertainty in a specific subset of questions, which is simple to enumerate. My review is targeted rather than distrustful. I get to trust most of questions and spend my time examining the handful that actually need it. Concretely: what used to take a day and a half of careful reading now takes about an hour of agent time (plus the compute cost, which is genuinely negligible at this scale) and two to three hours of human review. That's a 3× to 5× speedup on the part of the work that can be accelerated, and meaningful ecosystem coverage that would otherwise require headcount I don't have. Research assistant, not leader</h2> I want to be honest about what this system is and isn't. It is a research assistant. It has meaningfully accelerated my ability to produce assessments across an ecosystem that would otherwise require either much more time or much more labour. It is not a replacement for human judgement, and given the current state of the art, I don't believe it can be. The questions where sources contradict one another and someone has to decide which to trust; where you need to understand that a registry was acquired by a company with a different security culture and factor that into how you read their documentation; where a new threat has emerged that isn't on the 52-question list yet… That all still requires a human who understands the domain, has read the security community's work on similar registries, and can integrate signals that aren't in any web-accessible document. The split that actually works, after four attempts at getting it right, is this: the agent handles the grind, but the hard calls remain mine. I think that's fine. "Research assistant, not leader" is an honest description of what AI tooling can do well in a domain that requires nuanced judgement. The temptation to overclaim—to pretend the agent is doing the research rather than accelerating the parts of research that benefit from being done at scale—is real, and the corrective is a small dose of deflation. This thing doesn't do my job. It makes me faster at the parts of my job that can be made faster, and it captures my slow-and-careful work in a form that speeds up the next round. That's enough. Closing: what generalises</h2> While the outcome is specific, the lessons aren't. They are, I hope, useful if you are building any kind of research or evaluation agent—or more generally, any system where an LLM has to produce answers a human will rely on. Mechanical failures and quality failures look similar from the outside but have entirely different causes. Separate them early, because the toolkits are different and the techniques don't transfer. A hung process wants a retry; a confabulation wants a prompt rule. If you're shipping more schema validation to fix confabulations, you are doing the wrong thing. Tools have defaults. Choose them by their instincts, not by their spec sheets. The same model can behave very differently in different harnesses, and if your harness's default instinct is fighting your problem you will spend most of your energy fighting the harness. Pick the one that matches your problem's shape. Operational and behavioural instructions are different jobs. Write them separately. Procedural rules want to be simple and unambiguous ("use jq</code>, don't parse JSON yourself"); behavioural rules want the kind of precision that comes from reviewing failures ("a quote must be content from the source, not a description of your search process"). If your prompt feels bloated, it may be because the two kinds of rule are tangled together and neither is doing its job well. Observability is not optional for agents. A structured runtime file (every search, every URL, every candidate quote, every decision) is not paranoia and it is not for debugging after the fact. It is the raw material you use to understand what the model actually does, which is what turns prompt refinement from guesswork into iteration. If you can't see what the agent did, you can't tell it to do better. Non-determinism is both antagonist and tool. Crush it where you need reproducibility: structured outputs, deterministic tools, runtime logs you can replay. Weaponise it where you need creativity: ask the same question three different ways, sample more phrasings than you think you need, let the model's tendency to riff work for you. Both at once, applied to different layers of the same system. The best human-in-the-loop designs capture the human's work as durable input to the next run. If your system relies on a human to resolve the cases the agent can't handle, the cost of that human's work should decrease over time as the system learns which kinds of cases it can't handle and pre-routes them. A system that makes you re-resolve the same problem on every run is a worse system than one that gets out of your way as you teach it. I came to this project expecting to build something clever and instead learnt several old engineering lessons by running straight into them. While few of the lessons were genuinely new, all of them were surprising, in context, because LLMs are new enough that we haven't yet mapped our existing engineering intuitions onto them correctly. You cannot code your way out of a quality problem. You cannot make a non-deterministic system deterministic by putting schemas around it. Observability beats sophistication. Small sharp tools beat frameworks. Write a runtime log. Measure twice, cut once. Notes</h2> ^{1 Sonatype, 10th Annual State of the Software Supply Chain Report, 2024. https://www.sonatype.com/state-of-the-software-supply-chain </div>}^{2 Supply-chain Levels for Software Artifacts (SLSA), v1.2. https://slsa.dev. "SLSA is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus." </div>}^{3 The Sigstore project. https://docs.sigstore.dev/about/overview/. An OpenSSF-hosted project for signing and verifying software artefacts. </div>}^{4 OpenSSF Scorecard. https://github.com/ossf/scorecard </div>}^{5 Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal, "Detecting hallucinations in large language models using semantic entropy," Nature 630, 19 June 2024. https://www.nature.com/articles/s41586-024-07421-0. "We develop entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations — confabulations — which are arbitrary and incorrect generations." </div> ^{6 Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang (OpenAI), "Why Language Models Hallucinate," arXiv:2509.04664, 4 September 2025. https://arxiv.org/abs/2509.04664 </div> ^{7 Michael Townsen Hicks, James Humphries, and Joe Slater, "ChatGPT is bullshit," Ethics and Information Technology 26:38, 8 June 2024. https://link.springer.com/article/10.1007/s10676-024-09775-5. "The falsehoods and overall activity of large language models is better understood as bullshit in the sense explored by Frankfurt... the models are in an important way indifferent to the truth of their outputs." </div> ^{8 Harry G. Frankfurt, On Bullshit (Princeton University Press, 2005; originally a 1986 essay). "He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose." </div> ^{9 Chip Huyen, "Data Distribution Shifts and Monitoring," 7 February 2022. https://huyenchip.com/2022/02/07/data-distribution-shifts-and-monitoring.html </div> ^{10 Hamel Husain and Shreya Shankar, "LLM Evals: Everything You Need to Know," January 2026. https://hamel.dev/blog/posts/evals-faq/ </div> ^{11 Eugene Yan, "An LLM-as-Judge Won't Save The Product — Fixing Your Process Will," April 2025. https://eugeneyan.com/writing/eval-process/ </div> ^{12 HumanLayer, "Skill Issue: Harness Engineering for Coding Agents," March 2026. https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents </div> ^{13 Erik Schluntz and Barry Zhang, "Building effective agents," Anthropic Engineering, 19 December 2024. https://www.anthropic.com/engineering/building-effective-agents </div> ^{14 Simon Willison, "I think 'agent' may finally have a widely enough agreed upon definition to be useful jargon now," 18 September 2025. https://simonw.substack.com/p/i-think-agent-may-finally-have-a </div> ^{15 Horace He et al. (Thinking Machines Lab), "Defeating Nondeterminism in LLM Inference," 10 September 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ </div> ^{16 Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan, "Precise Zero-Shot Dense Retrieval without Relevance Labels" (HyDE), arXiv:2212.10496, 2022. https://arxiv.org/abs/2212.10496 </div> ^{17 Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences," UIST 2024 / arXiv:2404.12272. https://arxiv.org/abs/2404.12272 </div> ^{18 Douglas G. Altman and J. Martin Bland, "Absence of evidence is not evidence of absence," BMJ 311:485, 19 August 1995. https://www.bmj.com/content/311/7003/485 </div> ^{19 Simon Willison, "The lethal trifecta for AI agents: private data, untrusted content, and external communication," 16 June 2025. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ </div> ^{20 Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz, "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," arXiv:2302.12173, February 2023. https://arxiv.org/abs/2302.12173 </div> ^{21 Artem Chaikin and Shivan Kaul Sahib, "Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet," Brave Blog, 20 August 2025. https://brave.com/blog/comet-prompt-injection/ </div> ^{22 Nono sandbox. https://github.com/always-further/nono. Capability-based sandbox using Landlock (Linux) and Seatbelt (macOS). </div> Easily keep your local git repo up to date 2024-07-12T00:00:00+00:00 (This post originally appeared on dev.to</a>) Here's how to keep your clone of a Git repository current with upstream quickly and efficiently. Just pop this snippet into the .git/config</code> of your local checkout and you're good to go (some assembly required). [remote "upstream"] url = <http url> fetch = +refs/heads/*:refs/remotes/upstream/* [alias] refresh = "!git checkout main && git fetch upstream && git merge upstream/main main"</code></pre> Run git refresh</code> before you branch locally, and just like magic, you'll avoid merge conflicts and other irritating side effects. It's a bit dirty, but it'll do the job in most cases. 😉 Want to know more about what's going on here? Great! First: why? A few of the open source projects I work on have very active repositories, and since they all favour trunk-based development, that means main</code> gets updated frequently. Furthermore, they all prefer pull requests to be proposed from clones, as humans don't touch the primary repository directly. This is a really common model, so nothing special; however I always forgot to rebase before branching, except when I didn't forget to rebase, because I'd forgotten how to rebase. (Git is fun!</a>) So instead of having to do a little dance every time, I figured: 💭 let the computer do the work. And while incantations are great, I'm cranky enough to want to know what it is the computer is doing on my behalf, so let's break it down: [remote "upstream"] url = <http url> fetch = +refs/heads/*:refs/remotes/upstream/*</code></pre> The [remote]</code> block defines a remote repository, i.e. a repo that isn't on your computer. Here I've given it an arbitrary name (upstream</code>). Arbitrary because it's only used in this configuration context, so go with your heart on this one. Inside that block there's url = <http url></code>, which—you guessed it—defines the URL where the remote repository lives. This could be a link to Codeberg</a>, GitHub, GitLab, or whatever y'all got going on. Next is fetch</code>, which literally tells git how to fetch updates from that remote repository. That's the easy part. What do those other bits mean? The +</code> tells git to "force" update the references. If you're familiar with Git, this means forcing the references even if the result is a non-fast-forward update. Remember at the top of the post when I said "It's a bit dirty"? This is what I'm talking about. But here's the thing: it's probably what you want anyway. 😇 The refs/heads/*</code> part just tells git to get all the branches from the remote repo. Again: it's probably what you want. Finally, :refs/remotes/upstream/*</code> specifies where those fetched branches are going to be stored locally. There's a whole different blog post to be written about just this, but the tl;dr is we want to keep the upstream branches separate from the local branches, because things get crazy graph-theory style if we don't. 😝 OK, now the part you've all been waiting for: the actual command! [alias] refresh = "!git checkout main && git fetch upstream && git merge upstream/main main"</code></pre> The [alias]</code> block defines, well, an alias. It's basically a shortcut that lets us run arbitrary commands or sequences (including, notably, shell commands) so that we don't have to write them out by hand each time. In this case, the alias is refresh</code>; the idea here being that we can type git refresh</code> on the shell and it runs the command sequence, thus simplifying our lives somewhat. And some is better than none. Also: remember at the top of the post when I said "It's a bit dirty"? This shell command is actually what I'm talking about. 🥸 Let's be legends… The !</code> right at the start of the sequence tells git to treat the whole thing as a shell command, so whatever comes next will just be executed like we typed it out. First we make sure that we're on the main</code> branch in our local repo by running git checkout main</code>. It's possible that your repo uses a different name for trunk ("master" was popular for a long time), but "main" is common. Then we git fetch upstream</code> to nab that fresh upstream data from the remote repository. Remember upstream</code>? It's the remote repository we defined previously. Ah what fun we had. Finally, we merge the changes from the main</code> branch of upstream</code> into the main</code> branch of our local repository. That's literally how to read git merge upstream/main main</code>. 🧠 And that's it! A couple of lines gets us a quick, reusable, fairly reliable method of keeping our local repo up to date. Hope that helps! Let me know! BONUS: You might also be wondering what those &&</code> are for in between each of the commands. The Very Correct Definition is that &&</code> is a "logical AND operator". What it's used for in this case is a way to chain a series of shell commands into a sequence, whereby the next command will only be called if the current one was successful. This way, we don't end up doing horrible things to our local repo if</del> when things go wrong. Presentations and videos 2020-11-17T00:00:00+00:00 Hi there! I'm Dan. Over the past few years I've done a fair number of presentations, talks, panels, and other appearances at technical conferences and shows. You can check out the complete list of content—including videos, slide decks, and links to additional resources where available—at https://speaking.dark.ca/ . Thanks, and as always, please enjoy responsibly! Observability as code: dashboards 2020-09-10T00:00:00+00:00 I recently received a really great question from a Datadog</a> customer that I'd like to share concerning Observability as Code—specifically managing dashboards via Terraform</a> in this case, but the question and answer apply more broadly. While we use Terraform for 100% of our Datadog monitors, we've found [that] developers are less likely to update dashboards when they see a mistake in the UI, because they need to open a branch, run a pipeline and merge it in. This is a lot of overhead for a quick dashboard fix. Do you have any thoughts on this? SRE Team Lead </blockquote> First, the standard disclaimer: There’s no one true way to do anything, and every recommendation or “best practice” out there needs to be weighed against the reality of the environment and organisation that you’re a part of. :) That said, there are a few ways to look at this. If you’re part of a small shop with only a handful of resources, then it might be entirely reasonable to manage everything by hand—not just the dashboards. Scale, therefore, is a big part of the equation—and that notion extends to both computers and people. If you’re part of a small team that works either in isolation, or on things which are so specific as to be unusable to others, then here again, it’s entirely reasonable to manage those dashboards by hand. Still, even here one could make an argument for treating them as code: it’s self-documenting (for values of documentation), and it serves as a backup mechanism for when one of those manual edits inevitably goes awry. As scale increases, so does the value of managing the dashboards as code. A prime example is repeatability. Many organisations end up deploying multiple iterations of very similar stacks. Creating the dashboards for these stacks by hand becomes tiresome very quickly. Furthermore, as the stack itself changes, having to go through each dashboard set to update them can be a nightmare (not to mention making sure that future stacks get the correct dashboard updates as well). Some organisations may have more tightly controlled environments than others, and traceability for modifications is important. Even in environments that aren’t particularly controlled, there may be a healthy peer review culture that, far from being a burden, is a huge asset in ensuring that even small changes have more than one set of eyes in order to ensure completeness and suitability. In these cases, having tight feedback loops and rapid CI/CD mechanisms is paramount—if engineers feel like they’re fighting heavyweight tooling for “small changes”, then that’s a problem that should be addressed beyond the issue of dashboards specifically. If you want to learn more about Datadog, Hashicorp Terraform, and Observability as Code, I invite you to check out our upcoming joint webinar</a> on the topic on 23 September at 12:00 UTC. Hope to see you there! Devopsdays Berlin 2018; recap 2018-09-18T00:00:00+00:00 The sixth edition of DevOpsDays Berlin</a> was held over the 12th and 13th of September 2018, and despite the fact that it's one of the most consistent, longest-running Devopsdays events on the calendar, this year was my first ever opportunity to attend. The tl;dr is that this was a very fine event, and the local organising team clearly has their act together (though they are actively recruiting</a> for new volunteers). The talks were not recorded, so for the benefit of those who couldn't make it, I've refined my notes from the talks on Day 2 (unfortunately I had to miss Day 1). Dirk, on trust</h3> The day started with a presentation from Dirk Lehmann</a> of SAP entitled "Trust as Foundation of DevOps". Dirk has been a DevOps practitioner and booster for years, and I've had the pleasure of seeing him speak a handful of times before - as usual, he didn't disappoint. The central thesis of this talk is right there in the title: that trust, that fickle, difficult, human emotional thing, is the fundamental element of DevOps. He proposed a few ways to think about trust beyond a philosophical concept, and introduced a way of measuring the level of trust within an organisation: by measuring speed. Fast decision making, execution, and turnaround are hallmarks of a trust-based organisation. In terms of fostering a trusting environment, he proposed three key elements: managing team size, cultivating diversity, and embracing "positive failure culture". Regarding team size, he looked to the seminal Mythical Man Month, as well as the work of evolutionary psychologist Robin Dunbar (see: Dunbar's Number). Dirk's conclusion? 15 people is the largest a team can get before it it breaks - and often it should be smaller. Concerning diversity, the take-away was that teams should organise around projects, products, or goals, instead of the traditional "business unit" (or work-type silos) that have been commonplace for much of the post-industrial era. Finally, citing Google's research that investing in psychological safety - more than anything else - was critical to making a team work, Dirk argued that embracing failure as a learning experience is paramount. Yuwei, on mentoring</h3> The next talk was by Yuwei Ren</a>, a Production Engineer at Facebook, entitled "Mentorship: From the receiving end". Despite the title, most of her talk was about mentorship from the perspective of the mentor, and it was chock full of good tips for anybody engaging in a mentoring relationship. For example, as a mentor, how do you get off to a good start with your mentee? Start by establishing career goals, and then ask "How can I help get you there?". Ensure that the expectations are set, and that outcomes are clearly defined by both parties. This is done by establishing a plan to achieve those goals, and setting up milestones along the way. During the process, schedule regular check-ins, and stick to them! Set the bar high, and along with those lofty expectations, make sure that both parties are accountable to each other. The mentor/mentee relationship is a big responsibility, and it must be treated with respect and commitment. Both parties must acknowledge that disagreements and misunderstanding are inevitable, so empathy is key. Assuming good intentions is important, but that's only the first part of the equation - both parties need to actively listen and respect each other both professionally and as human beings. Finally, the mentee needs to take responsibility for their career, actively seeking and providing feedback, and acting on the advice of the mentor. Ken, on DevOps transformations</h3> Next up was Ken Mugrage</a> of Thoughtworks (and fellow member of the DevOpsDays global core) with a talk entitled "Want to successfully adopt DevOps? Change Everything". His talk was very dense, and to be honest I didn't quite capture all of it. His four key points to successful DevOps adoption were: redefine words, change your organisation, change your architecture, and use CD to safely deploy more often. Within a team specifically, and an organisation in general, agreeing on specific definitions is critical. In terms of defining the word DevOps, Ken argued that DevOps should not define a toolset, nor a role, nor a team. Instead, DevOps is verbs: developING and operatING. If you need a noun, why not CAMS or CALMS (upon which much has been written</a> before). Ken then offered his interesting description of DevOps: A culture where people, regardless of title or background, work together to imagine, develop, deploy, and operate a system. </blockquote> Words are important, but action is critical, and the best way to implement CA(L)MS is to change how your organisation is, well, organised. Ken invoked Conway's Law</a> and by way of illustration, explained how just renaming the Ops Team to DevOps team isn't going to help (nor is any other action that just leaves silos in place). Teams need to be organised differently, which means that the whole organisation might need to be organised differently as well. Furthermore, the success of those teams must be measured by how they deliver business value (not just whether their software functions or not). Ken then moved into continuous delivery versus continuous deployment, and this is where things got fuzzy for me. Sponsors!</h3> Three sponsors had an opportunity to say a few words on stage: Chef</a>, Google Cloud</a>, and Quest</a>. I mention this because it's important to acknowledge the important role that sponsors play in the DevOpsDays series - without them, the events simply wouldn't be able to happen. Thanks, sponsors! Tiago, on Microsoft</h3> The final full presentation of the day was from Tiago Pascoal</a> from the Azure DevOps Customer Advisory Team, speaking about how a unit within the Microsoft empire experienced its own DevOps transformation. This one started slow, but once Tiago got going, it ended up being very interesting - so I'll skip right to the good stuff. Within the Azure DevOps unit, teams are assembled into 10 to 12 person groups, the composition of which is heterogeneous and organised around products (see: Dirk's argument above). These teams are self-forming, so everybody decides which team and/or manager they want to work with. Apparently, while 100% of the group gets a choice, typically less than 20% choose to change from their existing situation. (To me, this is wild, and I'd be very interested to try something like this out at some point in my career.) Tiago invoked Daniel Pink's popular book Drive, noting that need teams 3 things: autonomy, mastery, and purpose. He added another layer, which he referred to as "Alignment vs Autonomy". Autonomy is everything related to planning and execution, whereas alignment is the organisation, the team structure, and most importantly, the cadence of work. For example, they work in 3-week sprints, and at the end of the sprint, whatever is in the master branch gets deployed - so be ready. At an organisational level they track metrics; but NOT the classics like burndown, velocity, original estimate, completed hours, team capacity, or number of bugs found. Individual teams can track those if they wish, but at a high level there's only one metric that matters: impact. Are they delivering value? Speaking of measuring, in terms of code quality, Tiago explained the "Bug Cap": n * 5 = x</code> where n</code> is # of engineers, and x</code> is maximum bug limit. If the bug count exceeds the bug cap at any time, the team stops working on new features and moves immediately to stabilise. Neat! Ignites</h3> After lunch, it was time for everybody's favourite DevOpsDays tradition: the ignites! I was preparing for my own ignite so I missed the first one</a> (sorry). Daiany, on relaxing at speed</h3> This was an excellently-delivered lightning that told the (true?) story of two teams and their relative stress levels heading into the GDPR cut-off date. It was fun, light-hearted, and carried a strong message: when you're doing the DevOps at high velocities, even massive EU legislation can't stress you out! Well done, Daiany</a>. Daniel, on serverless</h3> I</a> did a lightning talk on serverless. It was fun. :) Thanks!</h3> First off, thanks to the local DevOpsDays Berlin volunteer group. Also, thanks (again) to the sponsors. Finally, thank-you to my employer, Datadog</a>, for sending me out there to speak. There are so many moving parts to make even a small conference happen. I hope to make it out next year! So long, and thanks for all the fish. 2016-11-15T00:00:00+00:00 I joined Mozilla</a> nearly five years ago, which was an interesting period culturally for the company - we were transitioning from being a scrappy, disorganised group of hackers into a proper corporation with a global reach. One of my first bugs was to audit our load balancers - another was to help fix the tap in the office beer keg. It was an interesting time, fraught with mistakes, but heady with the promise of opportunity. For the first few years, I was part of the WebOps team, and as such, got to work with a wide range of projects and people around the world. Every day was a trial by fire, and while it was often times difficult or frustrating, the lens of hindsight now allows me to understand how valuable it was as a learning experience. From there, I moved into Services Engineering, and had the opportunity to work on the Crash Stats</a> project - a humbling experience if ever there was. I was exposed to some brilliant technologists and learned more about resilient systems design</a> than I ever dreamed possible. I was also exposed to some great managers, and will remain eternally grateful for the mentorship I received both directly and indirectly. Finally, I landed with the Cloud Services and Operations teams, and even though I'd been with the company for a long time, it was like my first days all over again: new and complex systems, new and complex people. Speaking of people, I'd be remiss if I didn't say this: Mozilla is full of some of the finest human beings I've ever had the honour and pleasure of working with. The spirit, drive, and desire to do good is something that is more than rare in the Valley (or anywhere), and the importance of this aspect cannot be overstated. Mozilla has treated me well. I've travelled the world, honed my skills, and learned so much, both professionally and personally. It's been a wild, interesting, life-changing ride. Today, more than ever, the Internet needs Mozilla. Even though my personal journey is taking me in a different direction, the mission and manifesto</a> are more important than ever before. The show must go on. Much love to all, and thanks for everything. Genesis: Terraforming a New Home for Firefox Crash Reporter 2016-03-24T00:00:00+00:00 Last year, my esteemed colleague JP Schneider</a> and I were invited to keynote at a couple of conferences last year. We gave two variations on the same talk, entitled "Genesis: Terraforming a New Home for Firefox Crash Reporter", at each of Hashiconf 2015</a> and Velocity EU 2015</a> respectively. The blurb for these talks is as follows: Everyone loves talking about different stacks used in migrating applications to DevOps methodologies, but the most important and valuable change is not a technology stack; instead, it is the human stack. In this keynote address, we will cover the intersection of technology and humans, and walk the audience through a real life example of the move of Firefox crash reporter. The three engineers tasked with this had to build the plane while it was in the air, all without landing or crashing. As with many projects, hard deadlines and requirements made the team work through a lot of tough decisions and compromises while simultaneously training devs, product managers, managers, and other engineers in the new world of DevOps and Continuous Delivery. </blockquote> The talks were a lot of fun and were well received by both audiences. We keep things light (including images and quotes from Shakespeare to Mobb Deep) and we keep them honest (both our successes and failures). The Velocity talk, which at 25 minutes in length is the shorter of the two, is aimed at a more general audience; the Hashiconf talk is longer, and includes a lot more detail about the Hashicorp tools that we used to reach our goals. I hope you enjoy either or both of them. :) Handling extant resources in Terraform 2015-01-27T00:00:00+00:00 Terraform</a> is a Hashicorp tool which embraces the Infrastructure as Code model to manage a variety of platforms and services in today's modern, cloud-based Internet. It's still in development, but it already provides a wealth of useful functionality, notably with regards to Amazon and Digital Ocean interactions. The one thing it doesn't do, however, is manage pre-existing infrastructure very well. In this blog post we'll explore a way to integrate extant infra into a basic Terraform instance. Note that this post is current as of Terraform v0.3.6</a>. Hashicorp has hinted that future versions of Terraform will handle this problem in a more graceful way, so be sure to check those changelogs regularly. :) summary</h1> A full example and walk-through will follow; however, for those familiar with Terraform and just looking for the tl;dr, I got you covered. Declare a new, temporary resource in your Terraform plan that is nearly identical to the extant resource.</li> Apply</code> the plan, thus instantiating the temporary "twinned" resource and building a state file.</li> Alter the appropriate id</code> fields to be the same as the extant resource in both the state and config files.</li> Perform a refresh</code> which will populate the state file with the correct data for the declared extant resource.</li> Remove the temporary resource from AWS manually.</li> Voilà.</li> </ul> faster and more dangerous, please.</h2> Walking through the process and meticulously checking every step? Ain't nobody got time for that! Edit the state file and insert the resource directly - it's just JSON, after all.</li> </ul> examples</h1> In the examples below, the notation [...]</code> is used to indicate truncated output or data. Also note that the AWS cli tool is assumed to be configured and functional. S3</h2> The extant resource in this case is an S3 bucket called phrawzty-tftest-1422290325</code>. This resource is unknown to Terraform. $ aws s3 ls | grep tftest 2015-01-26 17:39:07 phrawzty-tftest-1422290325</code></pre> Declare the temporary twin in the Terraform config: resource "aws_s3_bucket" "phrawzty-tftest" { bucket = "phrawzty-tftest-1422353583" }</code></pre> Verify and prepare the plan: $ terraform plan -out=terratest.plan [...] Path: terratest.plan + aws_s3_bucket.phrawzty-tftest acl: "" => "private" bucket: "" => "phrawzty-tftest-1422353583"</code></pre> Apply the plan (this will create the twin): $ terraform apply ./terratest.plan [...] aws_s3_bucket.phrawzty-tftest: Creation complete Apply complete! Resources: 1 added, 0 changed, 0 destroyed. [...] State path: terraform.tfstate</code></pre> Verify that the both the extant and temporary resources exist: $ aws s3 ls | grep phrawzty-tftest 2015-01-26 17:39:07 phrawzty-tftest-1422290325 2015-01-27 11:14:09 phrawzty-tftest-1422353583</code></pre> Verify that Terraform is aware of the temporary resource: $ terraform show aws_s3_bucket.phrawzty-tftest: id = phrawzty-tftest-1422353583 acl = private bucket = phrawzty-tftest-1422353583</code></pre> Alter the config file: Insert the name of the extant resource in place of the temporary.</li> Strictly speaking this is not necessary, but it helps to keep things tidy.</li> </ul> resource "aws_s3_bucket" "phrawzty-tftest" { bucket = "phrawzty-tftest-1422290325" }</code></pre> Alter the state file: Insert the name (id</code>) of the extant resource in place of the temporary.</li> </ul> "resources": { "aws_s3_bucket.phrawzty-tftest": { "type": "aws_s3_bucket", "primary": { "id": "phrawzty-tftest-1422290325", "attributes": { "acl": "private", "bucket": "phrawzty-tftest-1422290325", "id": "phrawzty-tftest-1422290325" } } } }</code></pre> Refresh the Terraform state (note the ID): $ terraform refresh aws_s3_bucket.phrawzty-tftest: Refreshing state... (ID: phrawzty-tftest-1422290325)</code></pre> Verify that Terraform is satisfied with the state: terraform plan Refreshing Terraform state prior to plan... aws_s3_bucket.phrawzty-tftest: Refreshing state... (ID: phrawzty-tftest-1422290325) No changes. Infrastructure is up-to-date. This means that Terraform could not detect any differences between your configuration and the real physical resources that exist. As a result, Terraform doesn't need to do anything.</code></pre> Remove the temporary resource: $ aws s3 rb s3://phrawzty-tftest-1422353583/ remove_bucket: s3://phrawzty-tftest-1422353583/</code></pre>S3, faster.</h2> For the sake of this example, the state file already contains an S3 resource called phrawzty-tftest-blah</code>. Add the "extant" resource directly to the state file. "resources": { [...] }, "aws_s3_bucket.phrawzty-tftest": { "type": "aws_s3_bucket", "primary": { "id": "phrawzty-tftest-1422290325", "attributes": { "acl": "private", "bucket": "phrawzty-tftest-1422290325", "id": "phrawzty-tftest-1422290325" } } }</code></pre> Refresh: $ terraform refresh aws_s3_bucket.phrawzty-tftest: Refreshing state... (ID: phrawzty-tftest-1422290325) aws_s3_bucket.phrawzty-tftest-blah: Refreshing state... (ID: phrawzty-tftest-blah)</code></pre> Verify: $ terraform show aws_s3_bucket.phrawzty-tftest: id = phrawzty-tftest-1422290325 acl = private bucket = phrawzty-tftest-1422290325 aws_s3_bucket.phrawzty-tftest-blah: id = phrawzty-tftest-blah acl = private bucket = phrawzty-tftest-blah</code></pre> That's that. Sysadvent 2014: What Sysadmins can learn from Community Management 2014-12-09T00:00:00+00:00 The annual Sysadvent series of blog posts is well under-way and this year features, for the first time ever, an entry authored collaboratively by myself and Jonathan Clarke</a> entitled « What System Administrators Can Learn From Community Management</a> ». This post was borne from notes collected over numerous Devops Paris meet-ups and other local community get-togethers, as well as from the personal experiences of both Jon and I in these respective domains. While the article is certainly valuable, it is by no means definitive, and my feeling is that it represents the beginning of the discussion - not the end. I'd love to hear from others who are involved in one or both of these areas, as there is so much more to explore concerning the interesting relationship and parallels between these interesting domains. Feel free to comment here, or better yet, hit me up on the Twitters</a>. For the sake of posterity, I will re-post the article here. Introduction</h3> At first blush it might not seem like there are any parallels between system administration and community management, but the two fields have much more overlap than you might think! Before we begin, let’s define community management briefly, and to do so, I’m going to lean on this this excellent post</a> by Deb Ng. It’s not marketing, nor social media, nor tech support, though these things can be involved. A community manager is a strategist, a content creator, and above all, a communicator. In this article, we will explore a handful of key principles from the community management world and see how they apply to system administration. It’s important to realise that both fields are concerned with systems, which is to say, “an assemblage or combination of things or parts forming a complex or unitary whole” (source</a>). In the case of sysadmins, one normally thinks of the relationship between nodes in a cluster, or clusters in a farm, or farms in a data centre, or - well, you get the idea. For community managers, one normally thinks of the relationship between members of a forum, or forums in a site, or sites in a community, or - well, here again, you get the idea. In both cases, the relationships between the individual elements of the whole can be complex, and often require specialised tools and years of experience to get a handle on. Managing groups at scale</h3> Scale is an important concept in community management. What works for managing a group of two people might not work for a group of 20 or 200 people. It’s generally easier to discuss a topic in a small group and achieve consensus when there are mutually-shared interests. Small groups can function directly together since the number of individual voices is low and the space needed for them to work is commensurate with their size. Logistically speaking, a small group can make use of pretty much any communication platform that happens to be convenient - a common list of people to CC when emailing and a Google document might well be enough. Contrast that with managing a community of, say, 50 people - that’s a big list of people to keep tabs on every time you need to write an email (not to mention the amount of noise that might get generated with an errant reply- all), and as anybody who’s ever tried to play a fast-paced game of multiplayer text editing knows, that’s a lot of cooks in the kitchen when it comes to a shared document. Even at this modest scale, some sort of more purpose-built tooling is required, such as a proper mailing list or forum site. This necessity is borne from two important effects of larger groups which, for our purposes, we can refer to as quantity and quality. Quantity is straightforward: more people means more conversations! A handful of emails per week is trivial for any one person to deal with, but multiply that by a factor of ten or 100, and the trickle becomes a raging torrent. The more information that’s being generated, the more difficult it becomes for any one member of the group to process it effectively. Quality refers to the nature of those conversations, and in particular, their level of diversity. Each person in the community will bring their own particular histories, viewpoints, cultural perspectives, and so forth to the group. Good tooling can make all the difference when trying to manage such a heterogenous system. A straightforward example of this is mature forum software that has conversational threads, thematic areas, and a robust back- end for gathering statistics and metrics about usage. From here it shouldn’t be difficult to make the mental leap to system administration. Two servers sitting on your desk are pretty straightforward to deal with - even if they’re totally different, there’s only two of them, so you probably don’t need any special infrastructure to manage them. Scale that up to a dozen servers, however, and the situation becomes much more complex. The dual effects of quantity and quality are instantly applicable. At even relatively small scales the importance of a coherent configuration management tool becomes evident: something that allows the different nodes in the system to be organised as operational or thematic groups, and with a robust back-end for gathering statistics and metrics about usage. Sound familiar? We’re going to come back to scale later, but for now, let’s move on to something a little bit more esoteric. Writing a mission statement</h3> I know what you’re thinking: “mission statement? That’s just corporate-speak nonsense!” And yes, that can certainly be the case, but it doesn’t have to be. In fact, a good mission statement is essential for any community, and explicitly stating it can help to crystallise the community in powerful ways. Most organic communities assemble around an initial idea or principle. While this shared element is enough to form the community, it’s rarely enough to maintain it, and almost certainly insufficient for real growth - the secret sustainability sauce is made up of the values that original germ represents. In other words, people come for the brilliant spark, but they stay for the ideas generated from the spark. A mission statement is a way to codify and present both. Sound a little high level? Consider that at some point in your life you’ve probably come across an open source project that was little more than a few hundred lines of code and a README. But something about this project made it different from the others. The documentation, ersatz though it might have been, clearly elucidated the problem and how this project acted as a (potential) solution. This simple addition to the repository made all the difference in the world because it helped you to understand how the project could fit your particular situation. Even without a deeper analysis of the code in question, your interest was piqued and your desire to learn more was set into motion. What happened there? Believe it or not, what captured your attention was, fundamentally, a mission statement. Most established companies have mission statements. These are organisation- wide screeds that may or may not have any direct relationship with the work that any one contributor effects within that organisation. Is it possible to bring the granularity level down a notch or two, and if so, what might the benefit be? Could it be useful to a system administration team? Imagine you’re just starting on a new project: a simple but powerful collection of scripts to manage virtual machines that’s going to make everyone’s day-to-day work much easier. Right now, even before you’ve begun writing any code, is the perfect time to write a mission statement. Start with something simple like “I’m fed up of typing in a dozen long and complicated commands every time I need to migrate a VM in our infrastructure, especially in the middle of the night, when on call. I intend this to be a small wrapper tool that can do normal migrations for us by typing in just one command. This should reduce the amount of time we waste on manual migrations and help to eliminate typing errors”. This brief, simple statement encapsulates both the problem and the solution, clearly defines the scope of the project, and describes both the initial idea germ and - critically - the parameters for success. There are numerous benefits to drafting your mission statement at the earliest possible stage. Crucially, by acting as a sort of combination compass and mold, it acts as a guide to keep the contributors focused (even if you’re the only one), which helps to nip things like feature creep in the bud straight away. Keep it short and don’t overthink it. Much like setting up some new test infrastructure, or working on a new program, the best thing is to start small and iterate over time as necessary. Finally, writing a mission statement should be easy; if you’re embarking on a project, then you know best what problem you’re trying to solve, and at those early stages is when you’ll have the most clarity about it. If for no one else, write the mission statement for your future self - a few weeks down the line you may have gone so deep you completely forgot the original problem you were trying to solve! A good mission statement can also be used to generate excitement which, as we’ll see below, can be a powerful tool for getting things done. Encouraging involvement</h3> A key principle of community management is actively encouraging contribution and involvement from the community. A good community manager will generate excitement and foster enthusiasm. Cultivating a positive emotional connection to the community will make members more loyal and encourage them to participate even more - a virtuous cycle that in turn makes the community even stronger. That said, while a general feeling of excitement is, like that initial idea germ, enough to create a spark, it’s insufficient to maintain the community. The excitement needs to be backed up by concrete measures to encourage active involvement. There is a fairly well-known philosophy in the open source world called “Release early, release often</a>” that was popularised by Eric S. Raymond in the late 90’s. The essence of the philosophy is that sharing versions of a software project early and often will ultimately improve the overall quality of the software. This happens because beta-testers and early adopters can have fun playing with new features while contributors can learn the codebase by fixing obvious bugs - that early feedback will help influence development in useful ways. In fact, as long as they’re not fatal, it can be helpful to leave some bugs in there deliberately! Providing low hanging fruit can enable an enthusiastic contributor to jump into the project by giving them the chance to fix that bug themselves. Looking at this from a systems perspective, this is basically a process with a feedback loop, and by releasing often the loop is accelerated. In other words, excitement is generated by rapid development, and the rapid development is encouraged by excitement - it’s almost too easy! As a system administrator, while you may certainly be leading up a project for which the rapid release model is applicable, that won’t always be the case. That’s no reason not to adopt this principle, however; by actively encouraging your co-workers to get involved, you help to build a trust relationship that will pay dividends down the line. Consider the development vs. operations silo for example - how nice would it be if the two groups actually worked together on projects right from the start? So how do you generate excitement? The easiest thing is to talk to your co- workers. Write a mission statement - or, better yet, draft one in collaboration with some potentially interested parties. Ask questions, discuss ideas, and continuously act on that feedback. Share design documents, proofs of concept, and initial implementations. Don’t be shy about unfinished releases or weird bugs, and don’t let perfection be the enemy of good - the important thing here is that both the features and the bugs are clearly explained. Whatever the scope of your next project, try sharing the idea with your co- workers (or whomever) before you get started and see how they react. Chances are, you’ll get their support a lot more easily that by showing them hundreds of lines of script that don’t yet solve the whole problem - and they probably won’t nitpick about your chosen scripting language or variable naming while reading the mission statement either. Of course for others to be able to fix bugs in your code, they need to know how to get that bug fix back into the code. Which brings us to our next topic"¦ Establishing community guidelines</h3> A simple search with the term “community guidelines” will reveal hundreds of thousands of examples from across the web. They encompass, either implicitly or explicitly, communities of just about every conceivable size on more or less every topic there is. It’s difficult to find a legitimate site that doesn’t have one - but, what are they? Simply stated, they are the basic tenets that frame the social contract between all of the members of the community, and they serve as ground rules to help that community to interact well. For example, an increasingly popular type of guideline is the conference code-of-conduct, which most conferences have now to help ensure that respectful and inclusive behaviour prevails. Responsible community management is, in many respects, predicated on the existence of community guidelines. It’s one of the most important tools that leaders can use to act decisively and consistently. When technical folk collaborate, be it in writing code or in administrating a server, we all tend to do things slightly differently from one another. How so? A simple question: where do you put custom binaries on a server? Right. Now, chances are you’re thinking “I don’t need to write out guidelines - we’re a tiny group”. You may well be right - your group probably can function just fine, for now; however, consider when a new person is joining your team. They will need to understand “how to do stuff around here”. Thinking further, the future could include a larger expansion, perhaps open sourcing a tool you wrote, or your company being acquired or merging with a larger one. In these cases, having planned ahead and written down the key points on how to work together could be a life saver - for both yourself and the new arrivals. As with everything, the key is to start simply: put a minimum viable set of guidelines together and then iterate as necessary. Some good starting points might include things like “all code must live in a given version control system.” That’s a simple, effective guideline that is both easy to follow and hard to misinterpret. Stipulate the conventions up front: paths on file systems, tabs vs. spaces for indentation, and so forth. That’s a great start. Next, explain how to contribute; this could be by sending a pull request on GitHub, a patch by email, or whatever. The important thing is to set up a consistent framework. Elucidate expectations for responses to contributions, too - it’s better to say something like “I only review patches on Mondays, so please be patient” than to leave an enthusiastic first time contributor hanging for six days. Examples of such standards documents (that have open source licenses) include Google’s style guide</a>, or for a more simple example, Rudder’s technical guidelines</a>. These contribution guidelines obviously tie into lowering the barrier to entry to your project too (see previous section). The underlying goal is to facilitate collaboration by avoiding pointless debates about which indentation style to use, when the project could be accepting contributions and moving forwards. There are no hard and fast rules governing the content or structure here, but as a general rule the idea is to be concise - use clear, brief language that expresses the core principles accurately. Similar guidelines can, and should, be drawn out for social interaction. Don’t fall into the trap of defining complex rules that are hard to follow and, most importantly, assume good intentions by default. Great relationships are built on trust, and the best way to get trust is to give it in the first place. There is really no one-size-fits-all set of rules and each community will grow around the shared values that have particular meaning to them; however, it is very important to point out what is not OK. Again, exact details here depend on who you are as a community but make sure that as a bare minimum you explicitly forbid any kind of discriminatory behaviour or harassment. Last but not least, remember that as one of your community’s leaders - however big or small that community is - you should be setting the example through action. Respect your own conventions. Welcome newcomers. Communicate clearly, consistently, and often. Be mindful that your words carry weight, and choose them deliberately and carefully when you write every email, blog post, or README. Resources</h3> The web is full of resources about community management - if you’re curious to learn more, some good starting points (for system administrators) include the Producing Open Source Software</a> book, Eric S. Raymond’s The Cathedral and The Bazaar</a>, and The Mozilla Manifesto</a>. Dawn Foster’s blog</a> also has some great insight on managing a community from a sysadmin perspective. Devopsdays Belgium 2014; recap 2014-11-04T00:00:00+00:00 Devopsdays Belgium 2014</a> was held over 27 and 28 October in the charming town of Ghent (or Gent, or Gand), Belgium. This event marked the five-year anniversary of the Devopsdays series of conferences - and with such an important milestone in play, expectations were running high. I'm happy to report that those expectations were met. First off, the basic details: The Devopsdays format is half-day presentations and half-day open spaces (the programme is available here</a>).</li> All of the presentations were recorded by BMC</a> and are available on Ustream</a> for your viewing pleasure.</li> Perusing #devopsdays on Twitter</a> is probably the fastest way to tap into the over-mind of the conference attendees, both in real-time and in retrospect.</li> The popular Hangops</a> show/podcast did a recap of the event</a> featuring speakers, attendees, and even Patrick Debois himself (if briefly).</li> </ul> Looking at the programme, it's interesting to note that there were no strictly-technical talks (the same was true, more or less, for the ignites as well). Broadly speaking, the topics ranged from management skills, to brain chemistry, and as far as a post-modern analysis of the Devops movement itself. Indeed, in terms of content, this was one of the most human-centric conferences I've ever attended. Five years ago this wouldn't have been the case and it is an indicator of just how far we've come as a community - in other words, huzzah, we've finally managed to stop talking about tools constantly. Concerning the presentations specifically, much has already been written, and I don't feel like I can add much more to the existing discourse. While I would recommend watching the videos themselves, for those looking for an executive summary of the two mornings, you could do worse than Carlos Sanchez's posts at InfoQ here</a>, here</a>, and here</a>. The best part of almost any conference, for me, is the so-called hallway track: those serendipitous encounters near the coffee machine or spontaneous introductions around a lunch table. In many ways, the open space</a> format is a formalisation of the hallway track, and one that works for everybody (not just people who are comfortable approaching strangers). In either case, when combined with the highly-local nature of Devopsdays, it made for a great opportunity to meet and speak with people whom I'd normally never encounter - the perfect environment for challenging discussions and generating fresh ideas. A great example of this environment came courtesy of Bridget Kromhout</a>, who (bravely) challenged the conference at large to use more inclusive language, especially when referring to groups within our industry (i.e. say "people" instead of "guys"). The subsequent conversation was civil, reasoned, thought-provoking, and - notably - itself inclusive, not only in terms of gender, but culture as well. This was interesting since it reminded everybody that, hey, different languages have different ways of dealing with this, and that while the problem exists everywhere, specific characteristics can vary wildly from region to region. I also had the pleasure of meeting Dave Mangot</a> who turned me on to something called the "1-1-1 model</a>" (as pioneered by Salesforce). Basically the idea is to donate 1% of profits, product, and time (respectively) to charitable ends. This got us thinking about how we could apply the model to Devopsdays, and by extension, tech conferences in general. Put another way, is it possible to leverage our industry's many social gatherings as a force for social good? I would submit that the answer is a resounding yes. For example, following the model, a given conference could: Donate 1% (or more, there's no upper limit!) of the profits to charity</a>.</li> Set aside a pool of tickets for people who otherwise wouldn't be able to afford to attend - or better yet, put together a grant programme to help for the cost of transport, accommodation, etc.</li> Facilitate an open space where attendees can volunteer to assemble care packages, sort clothing or food donations, or other types of manual activities along those lines.</li> </ul> Personally, I'd love to see Devopsdays adopt a model such as this one. Given the local, grassroots nature of the conference series, it seems like a perfect fit. Perhaps even the upcoming Devopsdays Paris, scheduled for April of 2015? Something to think about. As always, if you want to chat, feel free to leave a comment, or hit me up on the twitters</a>. Salut! DevOpsDays Paris 2013, the event that was 2013-04-23T00:00:00+00:00 I recently helped organise the inaugural Paris edition</a> of the popular DevOpsDays</a> series of conferences. It was a great event, and I'd like to share some of my thoughts and observations here. The DevOpsDays series of events is, as the name implies, centred around the "devops" movement, and is intended as a way to introduce people to this style of IT workflow, project, and people management. Since this is not related to a programming language, this conference falls outside of the normal sorts of events that Mozilla generally finds itself involved in. I believe this to be a very good thing, and I am proud to have represented Mozilla</a> both as a sponsor of the event, and as a European devops community member. The audience was mostly French in composition with a not-insignificant number of attendees from both Francophone and non-Francophone European countries. Said audience was a healthy mix of developers, IT operations, and managers across a wide spectrum of company sizes, types, and even industries; frankly, I was impressed at how broad the composition was, and it was refreshing to see interest in the devops movement from such a wide group. Though the event was held in Paris, all of the talks (with the exception of some of the ignites) were done in English. This was by design - a internally contentious decision that, in my opinion, ultimately proved itself to be the correct one. The open spaces during the afternoon were in a mix of English and French in order to ensure that everybody could participate equally. Concerning the open spaces, we weren't sure if the format would work here in France, but they were a smash success! Everybody seemed to really enjoy the format as a platform for discussion, debate, and idea-generation. I'd wager that for many of the attendees, it was the first time they'd ever been exposed to such a thing, and my hope is that they can bring the format to others in the future. Since devops is so new to France, the majority of the presentations themselves were entry-level, and thus not particularly interesting to me directly. That said, there were two presentations that really stood out (and would have held their own even at a more "advanced" event): "CustomerOps</a>" by Alexis Lê-Quôc</a>, and "Map & Territory</a>" by Pierre-Yves Ritschard</a>. Alexis' presentation on "CustomerOps" centred around the concept of providing customer support using engineering principles - and, indeed, delivered by engineers themselves. This really hit home for me because in Mozilla IT/Ops, we're not only the people who build and provide technical infrastructure, but are also the people who provide direct support to the consumers of that infrastructure - a situation that is absolutely not a given in many other companies (i.e. the admins and the customer reps are not the same people). Alexis illustrated the importance of communication, and how to measure success (read: customer satisfaction) in meaningful ways. Pierre-Yves' presentation was based on a very interesting philosophical conjecture: that our mental model of the world is not the same as the reality said model attempts to describe. Put another way, a map isn't actually land, it's a representation of the territory it describes (hence the title). Therefore, the most valuable models are the ones that can describe reality in useful ways, and it's in defining "usefulness" that the real effort must be made. In a more applicable sense his thesis was simple: identify your "key metrics" - the numbers which literally describe the success or failure of your business - and make sure you are collecting, analysing, and modelling them above all. Every other metric is either secondary or potentially uninteresting in the first place. Personally, I spent a lot of time mingling with the attendees, talking about Mozilla, our projects, and our mission. Generally speaking, the first question was, "Can I have one of those Firefox stickers?", but the second question was, "When can I get my hands on a FirefoxOS phone</a>?" As usual, everybody wanted to see one, and (unfortunately), as usual, I didn't have one to show them. The more events that I attend on behalf of Mozilla, the more I realise that continues to be a wasted opportunity to promote our most exciting new project. I'll have to work on this going forward. Of course, since this was a devops-related event, people were also very curious about if and how Mozilla is implementing devops internally. The overarching theme of devops is communication, so this event was an excellent opportunity to talk about IT at Mozilla</a>, and to promote not only our successes, but dig into our failures as well. This sort of interaction is vital in order to avoid stagnation. In summary, it was a fine showing for our first Parisian event, and I am looking forward to the next edition. Hopefully I'll see you there! hangops, european edition 2013-04-05T00:00:00+00:00 Today roidrage</a> and I</a> hosted the inaugural European edition</a> of the popular Hangops</a> podcast / hangout sessions. Hangops.eu (as we've taking to calling it) is functionally the same its North American counterpart - with the exception that "our" 11:00 is that of Western Europe, not California. ;) If you're not familiar with the Hangops, now's the time for you to get with the programme. The format is simple : get a group of talkative Ops people together on Google Hangouts, and see what happens ! Sometimes there are moderated talks, other times there are special guests, but the basic idea never changes - it's fundamentally a chance for nerds to present ideas, debate topics, learn a little something, and have a good time. The sessions are simulcast to Youtube, so if you don't want to join the hangout itself, you're invited to listen and participate in the IRC channel (freenode/#hangops) at your leisure. Hope to see you out next time! HAProxy Puppet module (phrawzty remix) 2013-03-18T00:00:00+00:00 As part of a big Logstash</a> project at Mozilla (more on that to come), I was looking for an HAProxy</a> module for Puppet, stumbling across the official Puppetlabs module</a> in the process. I'm told that this module works fairly well, with the caveat that it sometimes outputs poorly-formatted configuration files (due to a manifestly buggy implementation of concat</a>). Furthermore, the module more or less requires storeconfigs</a>, which we do not use in our primary Puppet system. Long story short, while I never ended up using HAProxy as part of the project, I did remix the official module to solve both of the aforementioned issues. From the README : This module is based on Puppetlabs' official HAProxy module; however, it has been "remixed" for use at Mozilla. There are two major areas where the original module has been changed : Storeconfigs, while present, are no longer required.</li> The "listen" stanza format has been abandoned in favour of a frontend / backend style configuration.</li> </ul> A very simple configuration to proxy unrelated Redis nodes : class { 'haproxy': } haproxy::frontend { 'in_redis': ipaddress => $::ipaddress, ports => '6379', default_backend => 'out_redis', options => { 'balance' => 'roundrobin' } } haproxy::backend { 'out_redis': listening_service => 'redis', server_names => ['node01', 'node02'], ipaddresses => ['node01.redis.server.foo', 'node02.redis.server.foo'], ports => '6379', options => 'check' }</code></pre> If that sounds interesting to you, the module is available on my puppetlabs-haproxy repo</a> on Github. Pull requests welcome ! LCA, the event that was 2013-03-11T00:00:00+00:00 Linux Conf Australia 2013</a>, (alias linux.conf.au, alias LCA) was an amazing experience, and it's difficult to summarise it succinctly, but there are definitely some highlights and important take-aways that I'd like to make special note of. My fundamental reason for attending was that Ben Kero</a> and I were invited to do a talk regarding Puppet</a> (an important infrastructural tool that we use at Mozilla). The talk, like all of the presentations given by Mozillians at LCA this year (and there were more than a few), was very well-received, and delivered to a packed auditorium - in fact, people had to be turned away at the door beforehand ! Since our talk bumped up against the lunch period, we had occasion to stay in the space and launch an informal panel discussion that included other Mozillians as well as representatives from a number of other Open Source companies and organisation. We engaged on a variety of topics, ranging from IT-centric to questions about the future of the web, and the importance of open standards and market competition. It was only when the organisers forced us out that the auditorium was finally cleared. Interestingly, from the moment I arrived in Australia, and through until the very last day, I found myself acting as an ambassador for Mozilla. For example, a day before the conference even started, I was part of an informal debate concerning the future of mobile and the importance of FirefoxOS within it. Other participants included a Ubuntu employee and a number of hardware hackers who - as it turned out - were already trying to port FirefoxOS to other types of phones. Another example even occurred at a local cricket match. I was wearing my Firefox T-shirt (which usually generates interest), and ended up doing a some impromptu demos of Firefox for Android to a couple of different groups - the highly satisfying result was that a handful of people downloaded and installed it on the spot ! These, and more, are all opportunities to engage people about Mozilla, and in many cases, to introduce our values and mission to entirely new audiences. During this and other conferences, I've found that even long-time users of Firefox and Thunderbird (for example) are often not even aware of who and what Mozilla is, and what we're about. I try my best to be a good ambassador, and I'm proud to represent the organisation wherever, and whenever I can. From a personal perspective, the conference had two major benefits: it was an excellent learning experience, and a fantastic opportunity to spend time in-person with a number of my co-workers in the IT/Ops team. Of note, I had the pleasure of attending a number of talks and presentations about technology that we use in our environment that have had a direct impact in my work-flow already. Beyond the directly-applicable benefits, I was particularly inspired by a presentation about programming for embedded devices</a> (shout-out to Bunnie Huang</a>) which, due to peculiar technical constraints, requires a very precise and measured approach to development. These ideals could - and should - be carried through outside of the embedded world. In summary, LCA 2013 was an excellent opportunity to learn, teach, and engage the public concerning technology in specific, Mozilla in particular, and open source in general. I'm already looking forward to next year ! How to use Puppet like an Adult 2013-02-15T00:00:00+00:00 Hello friends, Recently, Ben Kero</a> (a fellow Mozillian) and I were invited to present a talk at Linux Conf Australia</a>. To say that we were excited about presenting at one of the best Libre / Open Source conferences in the world is an understatement. We knew that we'd have to bring our A-game, and in all modesty, I like to think that we did. If you were there in person, I'd like to personally thank you for coming out, and if you couldn't make it, that's ok - the organisers have made many videos from the 2013 LCA available online</a>, including ours, entitled "How to use Puppet like an Adult</a>". We cover a variety of topics, including parametrisation, how to select good pre-built modules, and how you can build eco-systems around Puppet itself. Please feel free to drop us a line, either on Twitter</a> or here on the blog. Thanks ! Puppet at Mozilla, the podcast 2012-09-20T00:00:00+00:00 Hello everybody ! If you're interested in learning a little about Puppet at Mozilla, I highly recommend that you check out Puppet Podcast #7</a>, where Brandan Burton (@solarce</a>) and I (@phrawzty</a>) talk with Puppet Labs' Mike Stahnke (@stahnma</a>) about just that topic. It was our first podcast together, and frankly, it was more difficult that I thought it'd be. That said, it was a great experience, and I hope to do it again sometime. Hope you enjoy it ! quickly generate an encrypted password 2012-02-12T00:00:00+00:00 Hi everybody ! Here's a quick method for generating encrypted passwords that are suitable for things like /etc/passwd</code> . I realise that this isn't terribly complex, but honestly, I always forget how to do this until I actually need to do it - so here's a reminder for all of us. :) #!/bin/bash if [ "x$1" == 'x' ]; then echo "USAGE: $0 'password'" exit 1 fi # Get an md5sum of the password string; this is used for the SHA seed. md5=$( echo $1 | md5sum ) extract="${md5:2:8}" # Calculate the SHA hash of the password string using the extracted seed. mkpasswd -m SHA-512 "$1" "$extract" exit $?</code></pre> Elasticsearch backup strategies 2011-11-22T00:00:00+00:00 Update: This is an old blog post and is no longer relevant as of version 1.x of Elasticsearch. Now we can just use the snapshot</a> feature. Hello again! Today we're going to talk about backup strategies for Elasticsearch</a>. One popular way to make backups of ES requires the use of separate ES node, while another relies entirely on the underlying file system of a given set of ES nodes. The ES-based approach: Bring up an independent (receiving) ES node on a machine that has network access to the actual ES cluster.</li> Trigger a script to perform a full index import from the ES cluster to the receiving node.</li> Since the receiving node is unique, every shard will be represented on said node.</li> Shutdown the receiving node.</li> Preserve the /data/</code> directory from the receiving node.</li> </ul> The file system-based approach: Identify a quorum of nodes in the ES cluster.</li> Quorum is necessary in order to ensure that all of the shards are represented.</li> Trigger a script that will preserve the /data/</code> directory of each selected node.</li> </ul> At first glance the file system-based approach appears simpler - and it is - but it comes with some drawbacks, notably the fact that coherency is impossible to guarantee due to the amount of time required to preserve /data/</code> on each node. In other words, if data changes on node between the start and end times of the preservation mechanism, those changes may or may not be backed up. Furthermore, from an operational perspective, restoring nodes from individual shards may be problematic. The ES-based approach does not have the coherency problem; however, beyond the fact that it is more complex to implement and maintain, it is also more costly in terms of service delivery. The actual import process itself requires a large number of requests to be made to the cluster, and the resulting resource consumption on both the cluster nodes as well as the receiving node are non-trivial. On the other hand, having a single, coherent representation of every shard in one place may pay dividends during a restoration scenario. As is often the case, there is no one solution that is going to work for everybody all of the time - different environments have different needs, which call for different answers. That said, if your primary goal is a consistent, coherent, and complete backup that can be easily restored when necessary (and overhead be damned!), then the ES-based approach is clearly the superior of the two. import it !</h3> Regarding the ES-based approach, it may be helpful to take a look at a simple import script as an example. How about a quick and dirty Perl script (straight from the docs)</a> ? use ElasticSearch; my $local = ElasticSearch->new( servers => 'localhost:9200' ); my $remote = ElasticSearch->new( servers => 'cluster_member:9200', no_refresh => 1 ); my $source = $remote->scrolled_search( index => 'content', search_type => 'scan', scroll => '5m' ); $local->reindex(source=>$source);</code></pre> You'll want to replace the relevant elements with something sane for your environment, of course. As for preserving the resulting /data/ directory (in either method), I will leave that as an exercise to the reader, since there are simply too many equally relevant ways to go about it. It's worth noting that the import method doesn't need to be complex at all - in fact, it really shouldn't be, since complex backup schemes tend to have too many chances for failure than is necessary. Happy indexing! Send your logs to the cloud; Loggly vs. Papertrail 2011-11-17T00:00:00+00:00 N.B. This post is from 2011 - the landscape has changed since then... Centralised cloud-based logging. It sounds tasty - and it is - but who should you go with? Well, Loggly</a> and Papertrail</a> are the only games in town when it comes to the aforementioned service; the only other competitor in this space is Splunk Storm</a>, but their offering - well-pedigreed though it may be - is strictly in private beta at this time, and therefore cannot really be considered a valid option. The fact of the matter is that Loggly and Papertrail are, at a high level, functionally identical. They offer more or less the same bouquet of functionality, including alert triggers, aggregate visualisation, and even map reduce tools for data mining and reporting. Loggly has been around longer, and has a better track record for open-source involvement, meaning that the eco-system around their service is more mature; however, that doesn't mean that they are necessarily superior to Papertrail in terms of the actual service. My suggestion: If you're in a hurry, flip a coin and go with one or the other. If you have the time, you should go ahead and try both out for a bit; Papertrail has a 7-day free trial programme, and Loggly is free (in perpetuity) for sufficiently small amounts of data and retention (which is no problem if you're just poking around). I'm very interested in hearing about actual user experiences with either or both, so please don't hesitate to add a comment or drop me a line directly via the contact form. Edit: From @pyr</a> : « you can also consider @datadoghq</a> which has a different take on the issue but might fit the bill. » Edit 2: From the comments, there's also Logentries</a>, which I don't personally have any experience with, but which appears to offer a reasonably comprehensive offering as well. Heavyweight tilt : GitHub vs. Bitbucket 2011-11-08T00:00:00+00:00 When it comes to code hosting on The Internets today, GitHub</a> is absolutely the hottest, trendiest service going - but it's not alone. Right now, the primary direct competitor to GitHub is Bitbucket</a>, and choosing the best service for you or your company can be a less than obvious scenario - so let's break it down, shall we? GitHub is generally considered to be the most popular code hosting and collaboration site out there today. They have an excellent track record for innovation and evolution of their service, and they put their money where their mouth is, notably by promoting and releasing their own internal tools into the open source community. Their site offers a buffet of ever-improving facilities for collaborative activity, notably including an integrated issue tracker and excellent code comparison tools, among others. To be fair, not every feature has had the same level of care and attention paid to it, and as a result, some elements feel quite a bit more mature than others; however, again, they never stop trying to make things better. Bitbucket looks a lot like GitHub. That's a fact. I don't honestly know which one came first, but it's clear that today they're bouncing off of each other in terms of design, features, and functionality. You can more or less transpose your user experience between the two sites without missing too much of a beat, so for a casual user looking to contribute here and there, you get two learning curves for the price of one (nice). Bitbucket's pace of evolution is (perhaps) less blistering, but they too are capable of rolling out new and improved toys over time. let's get down to brass tacks</h3> Both services offer the same basic functionality, which is the ability to create an account, and associate that account with any number of publicly-accessible repositories; however, if you want a private repository, GitHub will make you pay for it, whereas BitBucket offers it gratis. There, as it is said, lies the rub. More on this later. One of the big differences between the two services lie in their respective origins: GitHub remains an independent start-up, whereas Bitbucket (although once independent) was acquired by - and is now strongly associated with - Atlassian (of JIRA fame). It is my opinion that this affects the cultural make-up of Bitbucket in subtle ways, leading to a more corporate take on development, deployment, and importantly, community relations and involvement. Take a look at their respective blogs (go ahead, I'll wait). A quick scan of the past few months from each blog will reveal some important differences: GitHub's release schedule is more aggressive, with improvements and new features coming more regularly, whereas Bitbucket places greater emphasis on their tight integration with JIRA, Jenkins, and other industry tools.</li> Bitbucket advertises paid services and software on their blog, whereas GitHub advertises open source projects.</li> Bitbucket's blog has one recent author, whereas GitHub's blog has many recent authors.</li> GitHub hosts more community events (notably drinkups, heh) over a greater geographic area than Bitbucket (and their posts have more community response overall).</li> </ul> Also, check out GitHub's "about us</a>" page - brogrammers</a> abound! I'd compare the group to Bitbucket, but as it so happens, they don't have an analogous page. Previously I mentioned that GitHub would like you to pay for private repositories. This is obviously part of their revenue scheme (and who can blame them for wanting to get that cheese?), but it also has the side-effect of making people choose to willingly host their projects publicly. This has ended up creating a (very) large community of active participants representing a variety of languages and interests, which in turn results in more projects, and so on and so forth. This feedback loop is interesting since it auto-builds popularity: as more people use it, the more people will use it. These observations are, in no way, objective statements of the superiority of one platform over the other - they are, however, indicative of cultural differences between the two companies. This is (or, at least, should be) a non-trivial element when deciding which service is right for you or your organisation. For example, I'm a beer-drinking open source veteran that works in start-ups and small companies, so culturally my preferences are different than those of a suit-wearing system architect, working for a thousand-person consulting firm. One isn't necessarily better than the other - they're just not the same (and that's OK). but wait, there's more</h3> Alright, here comes the shocker: for paid services (i.e. private repositories), GitHub is much more expensive than Bitbucket. As in nowhere near the same price. At all. How can this be? Well, I'm not privy to the financials of either company (if I were, I doubt I'd have written this post), but hey, the money for all those great open source projects, drinkups, and (bluntly) salaries have to come from somewhere - and while Bitbucket has Atlassian's pockets backing them, GitHub has to stand on their own successes, and live with their own failures. The two services are not dissimilar technically speaking, so it's really up to you to decide which culture is better suited for your project. Do you just need a spot to put your private project, that you program alone, isolated from the greater Internet? BitBucket. Do you have a public project that you'd like other people to discover, hack on together, and build a community around? GitHub. As for paid services, well I suppose that comes down to whether you want to pay extra to support what GitHub is doing or not. Now, let's be fair, for a lot of companies, "culture" is an irrelevant factor in their purchasing department - cost is the only concern. Fair enough. But let's say you've got a team of developers, all of whom already have their own projects on GitHub, are familiar with the tools and processes, and have a network of fellow hackers built-in and ready to go. In that case, perhaps culture is worth something after all. Improvements in Cassandra 1.0, briefly stated 2011-11-03T00:00:00+00:00 Datastax recently announced the availability of Cassandra 1.0 (stable), and along with that announcement, they made a series of blog posts (1</a>, 2</a>, 3</a>, 4</a>, 5</a>) about many of the great new features and improvements that the current version brings to the table. For those of you looking for an executive summary of those posts, you're in luck, cause I've got your back on this one. New multi-layer approach to compression that provides improvements to both write and (especially) read operations.</li> Said compression strategy also yields potentially significant disk space savings.</li> Leverages the JNA library</a> in order to provide in-memory caching; this procedure is optimised for garbage collection, resulting in a more efficient collection and a smaller overall footprint.</li> A much improved compaction strategy results in less costly compaction runs, improving overall performance on each node.</li> Fewer requests are made over the network, and said requests are smaller in size, improving overall performance across the cluster.</li> </ul> In short, 1.0 is a very significant, very important upgrade to 0.8 (et al.), and one which will likely bring it to the forefront of the hardcore big data / nosql scene at large. Nagios plugin to parse JSON from an HTTP response 2011-03-04T00:00:00+00:00 Update 2015-10-07: This plugin has evolved - please check the latest README</a> for up to date details. Hello all ! I wrote a plugin for Nagios that will parse JSON from an HTTP response. If that sounds interesting to you, feel free to check out my check_http_json repo</a> on Github. The plugin has been tested with Ruby 1.8.7 and 1.9.3. Pull requests welcome ! Usage: ./check_http_json.rb -u -e -w -c -h, --help Help info. -v, --verbose Additional human output. -u, --uri URI Target URI. Incompatible with -f. --user USERNAME HTTP basic authentication username. --pass PASSWORD HTTP basic authentication password. -f, --file PATH Target file. Incompatible with -u. -e, --element ELEMENT Desired element (ex. foo=>bar=>ish is foo.bar.ish). -E, --element_regex REGEX Desired element expressed as regular expression. -d, --delimiter CHARACTER Element delimiter (default is period). -w, --warn VALUE Warning threshold (integer). -c, --crit VALUE Critical threshold (integer). -r, --result STRING Expected string result. No need for -w or -c. -R, --result_regex REGEX Expected string result expressed as regular expression. No need for -w or -c. -W, --result_warn STRING Warning if element is [string]. -C is required. -C, --result_crit STRING Critical if element is [string]. -W is required. -t, --timeout SECONDS Wait before HTTP timeout.</code></pre> The --warn</code> and --crit</code> arguments conform to the Nagios threshold format guidelines</a>. If a simple result of either string or regular expression (-r</code> or -R</code>) is specified : A match is OK and anything else is CRIT.</li> The warn / crit thresholds will be ignored.</li> </ul> If the warn and crit results (-W</code> and -C</code>) are specified : A match is WARN or CRIT and anything else is OK.</li> The warn / crit thresholds will be ignored.</li> </ul> Note that (-r</code> or -R</code>) and (-W</code> and -C</code>) are mutually exclusive. Note also that the response must be pure JSON. Bad things happen if this isn't the case. How you choose to implement the plugin is, of course, up to you. Here's one suggestion: # check json from http define command{ command_name check_http_json-string command_line /etc/nagios3/plugins/check_http_json.rb -u 'http://$HOSTNAME$:$ARG1$/$ARG2$' -e '$ARG3$' -r '$ARG4$' } define command{ command_name check_http_json-int command_line /etc/nagios3/plugins/check_http_json.rb -u 'http://$HOSTNAME$:$ARG1$/$ARG2$' -e '$ARG3$' -w '$ARG4$' -c '$ARG5$' } # make use of http json check define service{ service_description elasticsearch-cluster-status check_command check_http_json-string!9200!_cluster/health!status!green } define service{ service_description elasticsearch-cluster-nodes check_command check_http_json-int!9200!_cluster/health!number_of_nodes!4:!3: }</code></pre> RabbitMQ plugin for Collectd 2011-02-24T00:00:00+00:00 Hello all, I wrote a rudimentary RabbitMQ plugin for Collectd. If that sounds interesting to you, feel free to take a look at my GitHub</a>. The plugin itself is written in Python and makes use of the Python plugin for Collectd. It will accept four options from the Collectd plugin configuration : Locations of binaries : RmqcBin = /usr/sbin/rabbitmqctl PmapBin = /usr/bin/pmap PidofBin = /bin/pidof</code></pre> Logging : Verbose = false</code></pre> It will attempt to gather the following information : From « rabbitmqctl list_queues » : messages memory consumser</code></pre> From « pmap » of « beam.smp » : memory mapped memory writeable/private (used) memory shared</code></pre> Props to Garret Heaton for inspiration and conceptual guidance from his « redis-collectd-plugin</a> ». how to use the Distributed Numeric Assignment (DNA) plug-in in 389 Directory Server 2010-04-20T00:00:00+00:00 Hello everybody ! Today's post is about the Distributed Numeric Assignment</a> (or « DNA » ) plug-in for the 389 Directory Server (also known as the Fedora, Red Hat, and CentOS Directory Servers). Although this plug-in has existed for quite some, there isn't a whole lot of documentation about how to implement it in a real-world scenario. I recently submitted some documentation to the maintainer of the 389 wiki</a>, but since i'm not sure how, when, or in what form that documentation will come to exist on their site, i thought i'd expand on it here as well. If you've made it this far, i'm going to assume that you're already familiar with the basics of LDAP, and already have an instance of Directory Server up and running - if not, i suggest you take a look through the official Red Hat documentation</a> in order to get you started. By way of some background, it is worth noting that my basic requirement was simply to have a centralised back-end for authenticating SSH logins to the various machines in our park. The actual numerical values for the UID and GID fields did not need to be the same, they simply needed to be both extant and unique for each user, with the further caveat that they should not collide with any existing values that might be defined locally on the machines. This is a very basic set of requirements, so it is an excellent starting point for our example. The first step is to activate the DNA plug-in via the console : [TAB] Servers and Applications Domain -> Server -> Server Group -> Directory Server [SECTION] Configuration Server -> Plug-ins -> Distributed Numeric Assignment [X] Enable plug-in Save</code></pre> The Directory Server needs to be restarted in order for the activation to take effect. This can either be done via the console, or via the command-line as normal. The next step is to define how DNA will interact with new user data ; this is different from configuring the plug-in itself, in that we will be setting up a layer in between the plug-in and the user data that will allow certain values to be generated automatically (which is, of course, the end goal of this exercise). Consider the following two LDIF snippets : # uids dn: cn=UID numbers,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config objectClass: top objectClass: extensibleObject cn: UID numbers dnatype: uidNumber dnamagicregen: 99999 dnafilter: (objectclass=posixAccount)) dnascope: dc=example,dc=com dnanextvalue: 1000 # gids dn: cn=GID numbers,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config objectClass: top objectClass: extensibleObject cn: GID numbers dnatype: gidNumber dnamagicregen: 99999 dnafilter: (|(objectclass=posixAccount)(objectclass=posixGroup)) dnascope: dc=example,dc=com dnanextvalue: 1000</code></pre> As you can see, they are nearly identical. This configuration activates the DNA magic-number functionality for the UID and GID fields as shown in the Posix attributes section of the console, though the values used may require further explanation. The only particular requirement for the magic number (specified by the « dnamagicregen » field) is that it be a value that cannot occur naturally, which is to say a value that would not be generated by the DNA plug-in, nor set manually at any time. The default value is « 0 », but since this is clearly a number with meaning on the average Posix system, i would recommend a suitably large number that is unlikely to ever be used, such as « 99999 ». Non-numerical values can technically be used too ; however, these will not be acceptable to the console, so unless you're using a third-party interface (or doing everything from the commandline), a numerical value must be used. The « dnanextvalue » field functionally indicates where the count will start from. As noted previously, in order to avoid collisions with existing local entries on the various machines, i chose a start point of « 1000 », which was more than acceptable in my environment. Once these two snippets are integrated via the commandline</a>, simply re-start the Directory Server (again), and you're good to go From now on, any time that a new user is created with the value « 99999 » entered into either (or both) of the UID and GID Posix fields, DNA will automagically generate real values as appropriate. Hope that helps - enjoy ! CPAN RPMs in RHEL / CentOS : generation, conflict, and solutions 2010-04-08T00:00:00+00:00 Hello all ! Today we're going to take a look at a somewhat obscure problem that - once encountered - can cause nothing but headaches for a system administrator. The problem relates to conflicts in CPAN RPM packages, and what can be done to work around the issue. If you've made it this far, i'm going to assume a couple of things : you're comfortable with RPMs and repositories, have worked with a .spec file before, and you know what Perl modules are. Good ? Ok, let's go. Edit : About a week after i posted this article, the pastebin i uploaded the examples to disappeared. Maybe it will come back - i don't know - but if not, sorry for the broken links... CPAN</a> is an enormous collection of Perl modules. If you've ever written a Perl script, there's a good chance you've used a module that - at one point or another - came from this archive. One of the really neat features of CPAN is the interactive manner</a> in which modules can be downloaded and installed from the archive using Perl right from the command line (frankly, if you're reading this post, there's a good chance you've used this feature, too). This is a fairly common way to install new modules and add functionality to your system, especially if you're coding for local use (i.e. on your personal box). It's useful, but it's not perfect, and one of the key areas where it starts to fail is scalability : if you've got a bunch of machines, and you need to SSH into each one to interactively install a CPAN module or two, it's going to be a hassle. Likewise, CPAN doesn't often find its way into the hearts and minds of enterprise Red Hat or CentOS environments, where the official policy is often to install software via RPM only (for support, administration, and sanity reasons, this is often the case). Luckily, some of the most commonly used CPAN modules exist as RPMs in the default repositories. Some, but not all (and not even « many ») - for this, there are other repositories available. Some examples : EPEL</a></li> Dag</a></li> RPMForge</a></li> Magnum Solutions</a></li> </ul> That last one - Magnum - is particularly interesting given the subject of our post today. From their info page : At Magnum we have a firm rule that all CPAN modules on our machines are installed from RPMs. The Fedora and Centos projects build RPMs for many CPAN modules, but there are always ones missing and the ones that are available often lag behind the most up to date versions. For that reason, we build a lot of RPMs of CPAN modules. And we don't want to keep that work to ourselves, so on these pages we make them available for anyone to download. </blockquote> Their RPMs are generated automagically using a great tool called « cpanspec », which does exactly what you think it does : given a CPAN tarball, it will generate a .spec file suitable for building an installable RPM. It is available in the standard repositories, and can be installed easily via YUM as normal, so go ahead and do that now. Ok, example time : say you needed HTML::Laundry, but after a quick peek through your repositories, it becomes readily apparent that an RPM is not available. Thanks to cpanspec, all is not lost : [build@host-119 ~]$ wget http://search.cpan.org/CPAN/authors/id/S/ST/STEVECOOK/HTML-Laundry-0.0103.tar.gz [build@host-119 ~]$ cpanspec --packager "build <build@domain.ext>" HTML-Laundry-0.0103.tar.gz</code></pre> We just downloaded the tarball right from the CPAN website, and ran cpanspec against it. The « --packager » argument simple defines the person who's generating the .spec, and doesn't necessarily have to be anything accurate. Go ahead and try it for yourself. Now take a look at the resulting .spec file (or on the a pastebin here</a>). As you can see, it fills in all the fields, including the critical (and often tricky-to-determine) « BuildRequires » and « Requires » items. Frankly, it's solid gold, and it has made the lives of CentOS / RHEL admins all over the world much easier. That said, it's not perfect, and there are times when you might run into problems. Actually, you may run into two problems in particular. The first is conflicts over ownership, which arises when multiple RPMs claim to be responsible for the same file (or files, or directories, or features, or whatever). The second is more nefarious : an RPM that writes files to the system without declaring ownership for them - a condition often referred to as « clobbering ». The former is irritating, but at least it's not destructive, unlike the latter, which can cause all manner of headaches. To illustrate these two problems, let's take a look at another example (this one being decidedly more real-world than that of Laundry above) : CGI.pm</a>. The .spec file</a> that is generated from this tarball is functional and correct, and we can build an installable RPM out of it, so at first all appears well. Again, go ahead and try for yourself - i'll wait. You may wish to capture the build output for review - otherwise, check the pastebin</a>. I'd like to draw your attention to the « Installing » lines. By trimming the « Installing /var/tmp/perl-CGI.pm.3.49-1-root-root » element from each of those lines, we can see the actual paths and files that this RPM will install to. Examples : /usr/lib/perl5/vendor_perl/5.8.8/CGI.pm /usr/lib/perl5/vendor_perl/5.8.8/CGI/Cookie.pm /usr/lib/perl5/vendor_perl/5.8.8/CGI/Util.pm /usr/share/man/man3/CGI.3pm /usr/share/man/man3/CGI::Pretty.3pm /usr/share/man/man3/CGI::Cookie.3pm</code></pre> At first glance this looks perfectly acceptable. But look what happens when we try to install the resulting RPM (clipped for brevity) : [root@host-119 build]# rpm -iv /usr/src/redhat/RPMS/noarch/perl-CGI.pm-3.49-1.noarch.rpm Preparing packages for installation... file /usr/share/man/man3/CGI.3pm.gz from install of perl-CGI.pm-3.49-1.noarch conflicts with file from package perl-5.8.8-27.el5.x86_64 file /usr/share/man/man3/CGI::Cookie.3pm.gz from install of perl-CGI.pm-3.49-1.noarch conflicts with file from package perl-5.8.8-27.el5.x86_64 file /usr/share/man/man3/CGI::Pretty.3pm.gz from install of perl-CGI.pm-3.49-1.noarch conflicts with file from package perl-5.8.8-27.el5.x86_64</code></pre> As it turns out, the Perl package that comes with RHEL / CentOS already contains CGI.pm. This is normal, since it's so popular, and is included as a convenience. Thus, RPM - in an attempt to preserve the coherence of the package management system - refuses to install overtop of the existing owned files. This is a fine illustration of the first of the two problems previously noted : conflicts over ownership. As i mentioned above, it's aggravating, but it's not a bug - it's a feature, and it's doing exactly what it's designed to do. Irritating, but not ultimately dire. If you look carefully, though, it's also an illustration of the second problem. Note the list of files that are conflicting. Look back to the list of files that the package contains - notice anything missing from the conflicts list ? That's right - the actual module files (*.pm) are not showing conflicts, which means they'd get overwritten without complaint by RPM. You might be thinking « who cares ? that's what i want » right now, but trust me, it's not what you want. Imagine this CGI package, with this version of CGI.pm gets installed, and then later you upgrade the Perl package - your CGI.pm files will get overwritten by the Perl package, because as far as RPM is concerned, Perl owns those files. All of a sudden, things break because you had scripts that relied on your particular version, but since you just upgraded Perl, you think (quite naturally) that the problem could be anywhere - where do you even start looking ? Imagine the headache if there are multiple administrators, multiple servers, multiple data centres, and multiple clients paying multiple dollars. No fun at all. So how can we upgrade CGI.pm, using an RPM, without running into these problems ? As is often the case, the answer is deceptively simple, but not immediately obvious. Ultimately what we want to accomplish is twofold : Avoid the man conflicts.</li> Ensure that the existing owned module files are not clobbered by our new package.</li> </ul> Concerning the man pages - and i'm going to be perfectly blunt here - the solution is to simply not install them, since, of course, they're already there. As for avoiding a clobbering condition, this requires a little bit of investigation into how Perl modules and libraries are stored on an RHEL / CentOS machine. Consider the following output : [root@host-119 ~]# ls -d /usr/lib64/perl5/* /usr/lib64/perl5/5.8.8 /usr/lib64/perl5/site_perl /usr/lib64/perl5/vendor_perl</code></pre> What's it all mean ? Well, the « 5.8.8 » directory is the default directory as defined by the Perl architecture, and is system and platform-agnostic, which is to say that it's (supposed to be) the same on every system. The « vendor_perl » directory contains everything that specific to RHEL / CentOS (the « vendor » of the distribution). As you may recall from the rpmbuild output above, this is where the RPM wants to install the modules (thus creating the clobbering condition). There's a third directory there, promisingly named « site_perl » ; as the name implies, this is where site-specific files are stored, which is to say items that are neither part of the default Perl architecture, nor part of the RHEL / CentOS distribution. As you've no doubt guessed by now, site_perl is where we're going to put our new modules. Luckily for us, the only thing that needs to be changed is the .spec file - and we even get a headstart, since cpanspec does most of the heavy lifting for us. Examining the .spec file</a> once more, we see the following lines of note (again, cut for brevity) : %build %{__perl} Makefile.PL INSTALLDIRS=vendor %files %{perl_vendorlib}/*</code></pre> These indicate that the target installation directory is that of the vendor, which is normally the case, and thus the default setting. Since we want to install to the site directory, we make the following changes : %build %{__perl} Makefile.PL INSTALLDIRS=site %files %{perl_sitelib}/*</code></pre> That solves our clobbering problem quite nicely, but what about the man files ? As i mentioned above, the idea is to simply avoid installing them altogether, but since they're generated automatically during the build process, how can we exclude them ? What i'm about to present is a bit of a hack, but it's absolutely effective, and ultimately quite clean : we delete them after they've been generated, and then don't declare them in the file list. Some items are already being potentially deleted by default, so let's go ahead and add our own line into the mix : find $RPM_BUILD_ROOT -depth -type d -exec rmdir {} 2>/dev/null ; # destroy manified man, man. find $RPM_BUILD_ROOT -type f -name '*.3pm' -exec rm -f {} ;</code></pre> This will look for all of the « manified » man files and just remove from the build tree. All that's left now is to remove them from the file list. This is as simple as deleting (or commenting out) their sole declaration : #%{_mandir}/man3/*</code></pre> Another option is to simply install use the « --excludedocs » argument when installing the RPM. I opted to remove the docs altogether in order to ensure that the package can be installed without errors by anyone else without needed to know about the argument requirement ahead of time (and to facilitate automated rollouts). What you'll end up with is a .spec file that looks like this</a>. Go ahead and build your RPM - it'll install without conflicts and without danger. This is a technique that can be used for other CPAN packages as well, so go ahead and install everything you've always wanted. workaround for slow shared folders in Virtualbox 3.x 2010-01-29T00:00:00+00:00 Happy 2010 fair readers ! I hope that all is well with you and yours. Let's get right to business : Virtualbox</a> has a feature that allows you to access the host OS's file system from the guest OS (shared folders</a>), which is super useful, but not exactly perfectly implemented. In particular, there are known, documented performance issues</a> in certain scenarios, such as when accessing a Linux host via a Windows guest (which, as you might imagine, is a pretty regular sort of activity). One common (?) workaround is to install and configure Samba on the Linux host, then access it from the Windows guest like one would access any network server. The problem here is that it requires that Samba be installed and configured, which can be a pain in the, well, you know. Furthermore, the connection will be treated like any other, and the traffic will travel up and down the network stack, which is fundamentally unnecessary since the data is, physically speaking, stored locally. Instead, here's another workaround, one that keeps things simple, and solves the performance problem : just map the shared folder to a local drive in the host OS. It's that easy. For those of us who aren't too familiar with the Windows explorer interface (me included, heh), there are tonnes of step by step instructions</a> available. For whatever reason (i suspect Netbios insanity), accessing the network share via a mapped drive manages to avoid whatever condition creates the lag problems, resulting in a rapid, efficient access to the underlying filesystem. Hope that helps - enjoy ! how to deal with broken time zones during a CentOS 5.3 kickstart 2009-12-01T00:00:00+00:00 Hello again fair readers ! Today's quick tip concerns the problem with missing time zones when deploying CentOS 5.3 (and some of the more recent Fedoras) in a kickstart environment. It's a known problem</a>, and unfortunately, since the source of the problem (an incomplete time zone data file) lies deep in the heart of the kickstart environment, fixing it directly is a distinct pain in the buttock region. There is, however, a workaround - and it's not even that messy ! The first step is to use a region that does exist, such as « Europe/Paris », which will satisfy the installer - then set the time zone to what you actually want after the fact in the « %post » section. So, in the top section of the kickstart file, we'll put : # set temporarily to avoid time zone bug during install timezone --utc Europe/Paris</code></pre> The « --utc » switch simply states that the system clock is in UTC, which is pretty standard these days, but ultimately optional. Next, in the %post section towards the end, we'll shoe horn our little hack fix into place : # fix faulty time zone setting mv /etc/sysconfig/clock /etc/sysconfig/clock.BAD sed 's@^ZONE="Europe/Paris"@ZONE="Etc/UTC"@' /etc/sysconfig/clock.BAD > /etc/sysconfig/clock /usr/sbin/tzdata-update</code></pre> So, what's going on there ? Let's break it down : In the first line, we're just backing up the original configuration file, to use in the next line...</li> The second line is the important one - this is the actual manipulation which will fix the faulty time zone, setting it to whatever we want. In this example « Etc/UTC » is used, but you can pick whatever is appropriate. The tool being used here is « sed », a non-interactive editor which dates back to the 1970's, and which is still used by system administrators around the world every day.</li> The command we're issuing to sed is between the single quotes - astute readers will notice that it's a regular expression</a>, but with @'s instead of the more usual /'s. In it, we simply state that the instance of « ZONE="Europe/Paris" » is to be replaced with « ZONE="Etc/UTC" ».</li> This change is to be made against the backup file, and outputted to the actual config.</li> </ul> </li> Finally, we run « tzdata-update » which, as you've no doubt guessed, updates the time zone data system-wide, based (in part) on the newly-corrected clock config.</li> </ul> And that, as they say, is that. Happy kickstarting, friends, and i'll see you next time ! load balancing, or, « how i learned to love a piranha » 2009-10-21T00:00:00+00:00 Hello again, everybody ! Today i thought that we'd take a look at a fun and useful topic of interest to many system administrators : load balancing & redundancy. Now, i know, it doesn't sound too exciting - but trust me, once you get your first mini-cluster set up, you'll never look at service management quite the same way again. It's not even that tough to set up, and you can get a basic setup going in almost no time at all, thanks to some great open source software that can be found in more or less any modern repository. First, as always, a little bit of theory. The most basic web server setup (for example), looks something like figure 001, below : [caption id="attachment_133" align="alignnone" width="243" caption="Figure 001"][/caption] As you can see, this is a functional setup, but it does have (at least) two major drawbacks : A critical failure on the web server means the service (i.e. the content being served) disappears along with it.</li> If the web server becomes overloaded, you may be forced to take the entire machine down to upgrade it (or just let your adoring public deal with a slow, unresponsive website, i suppose).</li> </ul> The solution to both of these problems forms the topic of this blog entry : load balancing. The idea is straightforward enough : by adding more than one web server, we can ensure that our service continues to be available even when a machine fails, and we can also spread the love, er, load, across multiple machines, thus increasing our overall efficiency. Nice ! batman and round robin</h3> Now, there are a couple of ways to go about this, one of which is called « Round Robin DNS » (or RRDNS), which is both very simple and moderately useful. DNS</a>, for those needing a refresher, is (in a nutshell) the way that human-readable hostnames get translated into machine-readable numbers. Generally speaking, hostnames are tied to IP addresses in a one-to-one or many-to-one fashion, such that when you type in a hostname, you get a single number back. For example : $ host www.dark.ca www.dark.ca has address 88.191.66.127</code></pre> In other words, when you type www.dark.ca into your browser, you get one particular machine on the Internet (as indicated by the address); however, it is also possible to set up a one-to-many relationship - this is the basis or RRDNS. A very common example is Google : $ host www.google.com www.google.com is an alias for www.l.google.com. www.l.google.com has address 74.125.39.99 www.l.google.com has address 74.125.39.103 www.l.google.com has address 74.125.39.104 www.l.google.com has address 74.125.39.105 www.l.google.com has address 74.125.39.106 www.l.google.com has address 74.125.39.147</code></pre> So what's going on here ? In essence, the Google administrators have created a situation whereby typing in www.google.com into your browser will get you one of a whole group of possibilities. In this way, each time you request some content from them, one of any number of machines will be responsible for delivering that service. (Now, to be fair, the reality of what's going on at Google is likely far more complex, but the premise is identical.) Your web browser will only get one answer back, which is more or less randomly provided by the DNS server, and that response is the machine you'll interact with. As you can see, this (sort of) satisfies our problem of resource usage, and it (sort of) addresses the problem resource failure. For those of you who are more visually inclined, please see figure 002 below : [caption id="attachment_137" align="alignnone" width="299" caption="Figure 002"][/caption] It's not perfect, but it is workable, and most of all, it's dead simple to set up - you just need to set your DNS configuration up and you're good to go (an exercise i leave to you, fair reader, as RRDNS is not really the focus of our discussion today). Thus, while RRDNS is a simple method for implementing a rudimentary load balancing infrastructure, it still has notable failings : The load balancing isn't systematic at all - by pure chance, one machine could end up getting hammered while others do very little, for example.</li> If a machine fails, there's a chance that the DNS response will contain the address of the downed machine. In other words, the chances of you getting the downed machine are 1 in X, where X is the number of possible responses to the DNS query. The odds get better (or worse, depending on how you look at it) as more machines fail.</li> A slightly more obscure problem is that of response caching : as a method of optimisation, many DNS systems, as well as software that interacts with DNS, will cache (hold on to) hostname lookups for variable lengths of time. This can invalidate the magic of RRDNS altogether...</li> </ul> another attack vector</h3> Another approach to the problem, and the one we'll be exploring in great depth in this article, is using a dedicated load balancing infrastructure, combining a handful of great open source tools and proven methodologies. First, however, some more theory. Our new approach to load balancing must propose both a solution to the original problems (critical failure & resource usage), as well as address and solve the drawbacks of RRDNS as noted above. Really, what we want is an intelligent (or, at least, systematic) distribution of load across multiple machines, and a way to ensure that requests don't get sent to downed machines by accident. It'd be nice if these functions were automated too, since the last thing an administrator wants to do is baby-sit racks of servers. What we'd like, in other words, could be represented by replacing the phrase « RRNDS » in figure 002 above, with the word « magic ». For now, let's imagine that this magic sits on a machine that we'll call « Load Balancer » (or LB, for short), and that this LB machine would have a similar conceptual relationship to the web servers as RRDNS does. Consider figure 003 : [caption id="attachment_143" align="alignnone" width="299" caption="Figure 003"][/caption] This is a basic way of thinking about what's going to happen. It looks a lot like figure 002, but there is a very important difference : instead of relying on the somewhat nebulous concept of DNS for our load balancing, we can now give that responsibility to a proper machine running and dedicated to the purpose. As you can imagine, this is already a huge improvement, since this opens the door to all sorts of additional features and possibilities that simply aren't possible with straight DNS. Another interesting aspect of this diagram is that, visually speaking, it would appear that the Internet cloud only « sees » one machine (the load balancer), even though there are a number of web servers behind it. This concept of having a single point of entry lies at the very core of our strategy - both figuratively and literally - as we'll soon discover In the here and now, however, we're still dealing with theory, and a solution based on « magic » is about as theoretical as it gets. Luckily for us though, magic is exactly what we're about to unleash - in the form of « Linux Virtual Server</a> », or « LVS » for short. From their homepage : The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer</a> running on the Linux operating system. The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single high-performance virtual server. [...] The Linux Virtual Server as an advanced load balancing solution can be used to build highly scalable and highly available network services, such as scalable web, cache, mail, ftp, media and VoIP services. The thing about LVS is that while it's not inherently complex, it is highly malleable, and this means you really do need to have a solid handle on exactly what you want to do, and how you want to do it, before you start playing around. Put another way, there are a myriad of ways to use LVS, but you'll only use one of them at a time, and picking the right methodology is important. The best way to do this is by building maps and really getting a solid feel for how the various components of the overall architecture relate to each other. Once you've got a good mental idea of what things should look like, actually configuring LVS is about as straightforward as it gets (no, really!). let's complicate the issue further, for science !</h3> Looking back to figure 003, we can see that our map includes the Internet, the Load Balancer, and some Web Servers. This is a pretty typical sort of setup, and thus, we can approach it from a few different ways. One of the decisions that needs to be made fairly early on, though, has more to do with topology and routing than LVS specifically : how, exactly, do the objects on the map relate to each other at a network level ? As always, there can be lots of answers to this question - each with their advantages and disadvantages - but ultimately we must pick only one. Since i value simplicity when it comes to technology, figure 004 describes a simple network topology : [caption id="attachment_147" align="alignnone" width="358" caption="figure 004"][/caption] Now, for those of you out there who may have some experience with LVS, you can see exactly where this is headed - for everybody else, this might not be what you were expecting at all. Let's take a look at some of the more obvious points : There are two load balancers.</li> The web servers are on the same network segment as the LBs.</li> Unlike the previous diagrams, the LBs do not appear to be « in between » the Internet and the web servers.</li> </ul> The first point is easy : there are two LBs for reasons of redundancy, as a single LB represents a single point of failure. In other words, if the LB stops working for whatever reason, all of your services behind it become functionally unavailable, thus, you really, really want to have another machine ready to go immediately following a failure. A little bit more explanation is required to explain the second and third points - but the short answer is two words : « Direct Routing</a> » (or DR for short). From the LVS wiki : Direct Routing [is] an IP load balancing technology implemented in LVS. It directly routes packets to backend server through rewriting MAC address of data frame with the MAC address of the selected backend server. It has the best scalability among all other methods because the overhead of rewriting MAC address is pretty low, but it requires that the load balancer and the backend servers (real servers) are in a physical network. If that sounds heavy, don't worry - figure 005 explains it in easy visual form : In a nutshell, requests get sent to the LB, which then passes it to the Web Server, who in turn responds directly to the client. It's fast, efficient, scalable, and easy to set up, with the only caveat being that the LBs and the machines they're balancing must be on the same network. As long as you're willing to accept that restriction, Direct Routing is an excellent choice - and it's the one we'll be exploring further today. a little less conversation, a little more action</h3> So with that in mind, let's get started. I'm going to be describing four machines in the following scenario. All four are identical off-the-shelf servers running CentOS 5.2 - nothing fancy here. The naming and numbering conventions are simple as well : [TABLE=2] You probably noticed the fifth item in this list, labelled « Virtual Web Server ». This represents our virtual, or clustered service, and is not a real machine. This will be explained in further detail later on - for now, let's go ahead and install the key software on both of the Load Balancer machines : [root@A01 etc]# yum install ipvsadm piranha httpd</code></pre> « ipvsadm » is, as you might have guessed, the administrative tool for « IPVS</a> », which is in turn an acronym for « Internet Protocol Virtual Server », which makes more sense when you say « IP-based Virtual Server » instead. As the name implies, IPVS is implemented at the IP level (which is more generically known as Layer-3 of the OSI model</a>), and is used to spread incoming connections to one IP address towards other IP addresses according to one of many pre-defined methods. It's the tool that allows us to control our new load balancing infrastructure, and is the key software component around which this entire exercise revolves. It is powerful, but sort of a pain to use, which brings us to the second item in the list : piranha. Piranha is a web-based tool (hence httpd, above) for administering LVS, and is effectively a front-end for ipvsadm. As installed in CentOS, however, the Piranha package contains not only the PHP pages that make up the interface, but also a handful of other tools of particular interest and usefulness that we'll take a look at as well. For now, let's continue with some basic setup and configuration. A quick word of warning : before starting « piranha-gui » (one of the services supplied by Piranha) up for the first time, it's important that both LBs have the same time set on them. You've probably already got NTP</a> installed and functioning, but if not, here's a hint : [root@A01 ~]# yum -y install ntp && ntpdate pool.ntp.org && chkconfig ntpd on && service start ntpd</code></pre> Moving right along, the next step is to define a login for the Piranha web interface : [root@A01 ~]# /usr/sbin/piranha-passwd</code></pre> You can define multiple logins if you like, but for now, one is certainly enough. Now, unless you plan to run your load balanced infrastructure on a completely internal network, you'll probably want to set up some basic restrictions on who can access the interface. Since the interface is served via an instance of Apache HTTPd</a>, all we have to do is set up a normal « .htaccess</a> » file. Now, a full breakdown of .htaccess (and, in particular, mod_access</a>) is outside of the scope of this document, but the simple jist is as follows : [root@A01 ~]# cat /etc/sysconfig/ha/web/secure/.htaccess Order deny,allow Deny from all # by default, deny from everybody Allow from 192.168.0 # requests from this network are allowed</code></pre> With those items out of the way, we can now activate piranha-gui : [root@A01 ~]# chkconfig piranha-gui on && service piranha-gui start</code></pre> Congratulations ! The interface is now running on port 3636, and can be accessed via your browser of choice - in the case of our example, it'd be « http://A01:3636/ ». The username for the web login is « piranha », and the password is the one we set above. Now that we're logged in, let's take a look at the interface in greater depth. look out - piranhas !</h3> The first screen - known as the « Control » page - is a summary of the current state of affairs. Since nothing is configured or even active, there isn't a lot to see right now. Moving on to the « Global Settings » tab, we have our first opportunity to start putting some settings into place : Primary server public IP : Put the IP address of the « primary » LB. In this example, we'll put the IP of A01.</li> Private server public IP : If we weren't using direct routing, this field would need a value. In our example, therefore, it should be empty.</li> Use network type : Direct Routing (of course!)</li> </ul> On to the « Redundancy » tab : Redundant server public IP : Put the IP address of the « secondary » LB. In this example, we'll put the IP of A02.</li> Syncdaemon : Optional and useful - but know that it requires additional configuration in order to make it work. This feature (which is relatively new to LVS) ensures that the state information (i.e. connections, etc..) are shared with the secondary in the event that a failover occurs. For more information, please see this page</a> from the LVS Howto.</li> It is not necessary, strictly speaking, so we can just leave it unchecked for now.</li> </ul> </li> </ul> Under the « Virtual Servers » tab, let's go ahead and click « Add », then select the new unconfigured entry and hit « Edit » : Name : This is an arbitrary identifier for a given clustered service. For our example, we'd put « WWW ».</li> Application Port : The port number for the service - HTTP runs on port 80, for example.</li> Protocol : TCP or UDP - this is normally TCP.</li> Virtual IP Address : This is the IP address of the virtual service (VIP), which you may recall from the table above. This is the IP address that clients will send requests to, regardless of the real IP addresses (RIP) of the real servers which are responsible for the service. In our example, we'd put 192.168.0.40 . Each service that you wish to cluster needs a unique « address : port » pairing. For example, 192.168.0.40:80 could be a web service, and 192.168.0.40:25 would likely be a mail service, but if you wanted to run another, separate web service, you'd need to assign a different virtual IP.</li> </ul> </li> Virtual IP Network Mask : Normally this is 255.255.255.255, indicating a single IP address (the Virtual IP Address above). You can actually cluster subnets, but this is outside of the scope of this tutorial.</li> </ul> </li> Device : The Virtual IP address needs to be assigned to a « virtual network interface</a> », which can be named more or less anything, but generally follows the format « ethN:X », where N is the physical device, and X is an incremental numeric identifier. For example, if your physical interface is « eth0 », and this is the first virtual interface, then it would be named « eth0:1 ». If and when you set up multiple virtual interfaces, it is important to not mix these up. Piranha has no facility for sanity checking these identifiers, so you may wish to track them yourself in a Google document</a> or something.</li> </ul> </li> Scheduling : There are a number of options here, and some are very different from one another. For the purposes of this exercise, we'll pick a simple, yet relatively effective scheduler called « Least-Connections ». This does exactly what it sounds like : when a new request is made to the virtual service, the LB will check to see how many connections are open to each of the real servers in the cluster, and then route the connection to the machine with the least connections. Congrats, you've now got load balancing !</li> </ul> </li> </ul> Finally, let's add some real servers into the cluster. From the « Edit » screen we're already in, click on the « Real Server » sub-tab. Name : This is the hostname of the real server. In our example, we'd put B01.</li> Address : The IP address of the real server. In our example, for B01, we'd put 192.168.0.38 .</li> Port : Generally speaking this can be left empty, as it will inherit the Port value defined in the virtual service (in this case, 80). A value would be required here if your real server is running the service on a different port than that specified in the virtual service ; if your real server is running a web service on port 8080 instead of 80, for example.</li> </ul> </li> Weight : Despite the name, this value is used in various different ways depending on which Scheduler you selected for the virtual service. In our example, however, this field is irrelevant, and can be left empty.</li> </ul> You can apply and add as many real servers as you like, one at a time, in this fashion. Go ahead and set up B02 (or whatever your equivalent is) now. If you're wondering when the secondary LB is going to be configured, well, wonder no longer : the future is now. Luckily, this step is very, very easy. From the secondary : [root@A02 ~]# scp root@A01:/etc/sysconfig/ha/lvs.conf /etc/sysconfig/ha/</code></pre>now is a good time to grab a beer</h3> Phew ! That was a lot of work. After consuming a suitable refreshment, let's move on to the final few steps. Earlier i mentioned that there were some other items that we'd need to learn about besides the Piranha interface - « Pulse</a> » is one such item. Pulse, as a tool, is in the same family as some other tools you may have heard of, such as « Heartbeat</a> », « Keepalived</a> », or « OpenAIS</a> ». The basic idea of all of these tools is simple : to provide a « failover</a> » facility between a group of two or more machines. In our example, our primary LB is the one that is normally active, but in the case that it fails for some reason, we'd like our secondary to click in and take over the responsibilities of the unavailable primary - this is what Pulse does. Each of the load balancers runs an instance of « pulse » (the executable, not the package), which behaves in this fashion : Each LB sends out a broadcast packet (a pulse, as it were) stating that they are alive. As long as the active LB (commonly the primary) continues to announce itself, everybody is happy and nothing changes.</li> If, however, the inactive LB (commonly the secondary) server notices that it hasn't seen any pulses from the active LB lately, it assumes that the active LB has failed.</li> The secondary, formerly inactive LB, then becomes active. This state is maintained until such a time as the primary starts announcing itself again, at which point the secondary demotes itself back to inactivity.</li> </ul> The difference between the active and the inactive server is actually very simple : the active server is the one with the virtual addresses assigned to it (remember those, from the Virtual Servers tab in Piranha?). Let's go ahead of start it up (on the primary LB first, then on the secondary) : [root@A01 ~]# chkconfig --add pulse [root@A01 ~]# service pulse start</code></pre>an internet ballgame drama - in 5 parts</h3> You may have noticed that we haven't even touched the « real » servers (i.e. the web servers) yet. Now is the time. As it so happens, there's only one major step that relates to the real servers, but it's a very, very important one : defining VIPs, and then ensuring that the web servers are OK with the routing voodoo that we're using to make this whole load balancing infrastructure work. The solution is simple, but the reason for the solution may not be immediately obvious - for that, we need to take a look at the IP layer of each packet (neat!). First, though, let's run through a series of little stories : Alice has a ball that she'd like to pass to Bob, so she tosses it his way.</li> Bob catches the ball, sees that it's from Alice, and throws it back at her. What great fun !</li> </ul> Now imagine that Alice and Bob are hanging out with a few hundred million of their closest friends - but they still want to play ball. Alice writes Bob's name on the ball, who then passes it to somebody else, and so forth.</li> Eventually the ball gets passed to Bob. Unfortunately for Bob, he has no idea where it came from, so he can't send it back.</li> </ul> The solution is obvious : Alice writes « From : Alice, To : Bob » on the ball, the passes it along.</li> Bob gets the ball, and switches the names around so that it says « From : Bob, To : Alice », and sends it back.</li> </ul> OK, so, those were some nice stories, but how do they apply to our Load Balancing setup ? As it turns out, all we need to do is throw in some tubes</a>, and we've described one of the basic functions of the Internet Protocol - that the source and destination IP addresses of a given packet are part of the IP layer of said packet. Let's complicate it by one more level : Alice prepares the ball as above, and send it flying.</li> Bob gets the ball, who's been avoiding Alice since things got weird at the bar last week-end, passes it along to Charles.</li> Charles - who's had a not-so-secret crush on Alice since high school - happily writes « From : Charles, To : Alice », and tosses it away.</li> Alice receives the ball, but much to her surprise, it's from Charles, and not Bob as she expected. Awkward !</li> </ul> With that last story in mind, let's take another look at figure 005 above (go ahead, i'll wait). Notice anything ? That's right - the original source sends their packet off, but then receives a response from a different machine than they expected. This does not work - it violates some basic rules about how communications are supposed to function on the Internet. For the thrilling conclusion - and a solution to the problem - let's return to our drama : As it turns out, Bob is a player : he gets so many balls from so many women that he needs to keep track of them all in a little notebook.</li> When Bob gets Alice's ball he passes it to Charles, then he records where it came from and who he gave it to in his notebook</li> Charles - in an attempt to get into Bob's circle of friends - agrees to write « From : Bob, To : Alice » on the ball, then sends it back.</li> Alice - expecting a ball from Bob - is happy to receive her Bob-signed spheroid.</li> Bob then gets another ball from Denise, passes it to Edward, and records this relationship as well.</li> Edward - a sycophant if ever there was - prepares the ball in the same fashion as Charles, and fires it back.</li> </ul> Of course, the more balls Bob has to deal with, the more helpers he can use to spread the work around. Now, as you've no doubt pieced together, Alice and Denise are any given sources on the Internet, Bob is our LB, and Charles & Edward are the web servers. Now, instead of writing people's names on balls, we should now make the mental leap to IP addresses in packets. With our tables of hostnames and addresses in mind, let's consider the following example : The source sends a request for a web page. The source IP is « 10.1.2.3 », and the destination IP is « 192.168.0.40 » (the VIP for WWW).</li> </ul> </li> The packet is sent to A01, which is currently active, and thus has the VIP for WWW assigned to it.</li> A01 then forwards the packet to B02 (by chance), which crafts a response packet. The RIP for B02 is « 192.168.0.39 », but instead of using that, the source IP is set to « 192.168.0.40 », and the destination is « 10.1.2.3 ».</li> </ul> </li> The source, expecting a response from « .40 », indeed receives a packet that appears to be from WWW. Done and done.</li> </ul> The theory is sound, but how can we implement this in practice ? As i said - it's simple ! We simply add a dummy interface to each of the web servers that has the same address as the VIP, which will allow the web servers to interact with packets properly. This is best done by creating a simple sysconfig entry on each of the web servers for the required dummy interface, as follows : [root@B01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-lo:0 # for VIP DEVICE=lo:0 IPADDR=192.168.0.40 NETMASK=255.255.255.255 BROADCAST=192.168.0.40 ONBOOT=yes NAME=vip0</code></pre> [root@B01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-lo:0 for VIP</h1> DEVICE=lo:0 IPADDR=192.168.0.34 NETMASK=255.255.255.255 BROADCAST=192.168.0.34 ONBOOT=yes NAMall together now The « lo » indicates that it's a « Loopback address</a> », which is best described by Wikipedia : Such an interface is assigned an address that can be accessed from management equipment over a network but is not assigned to any of the real interfaces on the device. This loopback address is also used for management datagrams, such as alarms, originating from the equipment. The property that makes this virtual interface special is that applications that use it will send or receive traffic using the address assigned to the virtual interface as opposed to the address on the physical interface through which the traffic passes. In other words, it's a fake IP that the machine can use to make packets anyways. Now, there is a known scenario in which a machine with a given loopback address will, in this particular situation, cause confusion on the network about which interface actually « owns » a given address. It has to do with ARP, and interested readers are encouraged to Google for « LVS ARP problem</a> » for more technical details - for now, let's just get right to the solution. On each of the real servers, we'll need to edit « sysctl.conf » : [root@B01 ~]# vim /etc/sysctl.conf</code></pre># this file already has stuff in it, so put this at the bottom net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2</code></pre> Now, restart sysctl</a> : [root@B01 ~]# sysctl -p</code></pre> That's it - problem solved. all together now !</h3> At this point we've now explored each key item that is necessary to make this whole front-end infrastructure work, but it is perhaps not quite clear how it all works together. So, let's take a step back for a moment and review : There are four servers : two are load balancers, and two are web servers. Of the two load balancers, only one is active at any given time ; the other is a backup.</li> </ul> </li> Every DNS entry for the sites on the web servers points to one actual IP address. This IP address is called the « Virtual IP ».</li> The VIP is claimed by the active load balancer, meaning that when a request is made for a given website, it goes to the active LB.</li> </ul> </li> The LB then re-directs the request to an actual web server. The re-direction can be random, or based on varying levels of logical decision making.</li> The web server will respond directly - the LB is not a proxy.</li> </ul> </li> </ul> Great ! Now, what software runs where, and why ? The load balancers use LVS in order to manage the relationship between VIPs and RIPs.</li> Pulse is used between the LBs in order to determine who is alive, and which one is active.</li> An optional (but useful) web interface to both LVS and Pulse comes in the form of Piranha, which runs on a dedicated instance of Apache HTTPd on port 3636.</li> </ul> And that, my friends, is that ! If you have any questions, feel free to comment below (remember to subscribe to the RSS feed for responses). Happy balancing ! oh, p.s., one last thing...</h4> In case you're wondering how to keep your LVS configuration file synchronised across both of the load balancers, one way to do it would be with a network-aware filesystem - POHMELFS</a>, for example. ;-) it's coming ! 2009-10-12T00:00:00+00:00 Hi all ! I'm currently working on a new Howto regarding Load Balancing. It's gonna be a good one, complete with graphical charts and everything - so stay tuned ! where to specify ethtool options in Fedora 2009-08-25T00:00:00+00:00 Hi everybody - here's a super-quick update for you concerning « ethtool</a> », and how to use it to set options in Fedora properly. Ethtool is a great little tool that can be used to configure all manner of network interface related settings - notably the speed and duplex of a card - on the fly and in real time. One of the most common situations where ethtool would be used is at boot time, especially for cards which are finnicky, or have buggy drivers, or poor software support, or.. well, you get the idea. Times were that if you needed to use ethtool to configure a NIC setting at boot time, you'd just stick the given command line into « rc.local</a> », or perhaps another runlevel script, and forget about it. The problem with this approach is (at least) twofold : Frankly, it's easy to forget about something like this, which makes future support / debugging of network issues more of a pain.</li> Anything that automatically modifies the runlevel script (such as updates to the parent package) may destroy your local edits.</li> </ul> In order to deal with these issues, and to standardise the implementation of the ethtool-at-boot technique, the Red Hat (and, thus, Fedora) maintainers introduced an option for defining ethtool parameters on a per-interface basis via the standard « sysconfig</a> » directory system. Now, this actually happened a number of years ago, but the implementation was poorly announced (and poorly documented at the time), and thus, even today a lot of users and administrators don't seem to know about it. Now, there's a very good chance that you already know this, but just to refresh your memory : in the sysconfig directory, there is another directory called « network-scripts</a> », which in turn contains a series of files named « ifcfg-eth? », where « ? » is a device number. Each network device has a configuration file associated with it ; for example, ifcfg-eth1 is the configuration file for the « eth1 » device. In order to specify the ethtool options for a given network interface, simply edit the associated configuration file, and add a « ETHTOOL_OPTS » line. For example : ETHTOOL_OPTS="autoneg off speed 100 duplex full"</code></pre> Now, whenever the network service initialises that interface, ethtool will be run with the specified options. Simple, easy, and best of all, standardised. What could be better ? (complex) partitioning in kickstart 2009-08-03T00:00:00+00:00 UPDATE: This article was written back in 2009. According to a commenter below</a>, Busybox has been replaced by Bash in RHEL 6; perhaps Fedora as well? Bonjour my geeky friends ! :) As you are likely aware, it is now summer-time here in the northern hemisphere, and thus, i've been spending as much time away from the computer as possible. That said, it's been a long time, i shouldn't have left you, without a strong beat to step to. Now, if you're not familiar with kickstarting, it's basically just a way to automate the installation of an operating environment on a machine - think hands-free installation. Anaconda is the OS installation tool used in Fedora, RedHat, and some other Linux OS's, and it can be used in a kickstart capacity. For those of you looking for an intro, i heavily suggest reading over the excellent documentation</a> at the Fedora project website. The kickstart configuration process could very easily be a couple of blog entries on its own (which i'll no doubt get around to in the future), but for now i want to touch on one particular aspect of it : complex partition schemes. how it is</h3> The current method for declaring partitions is relatively powerful, in that all manner of basic partitions, LVM components, and even RAID devices can be specified - but where it fails is in the creating of the actual partitions on the disk itself. The options that can be supplied to the partition keywords can make this clunky at best (and impossible at worst). A basic example of a partitioning scheme that requires nothing outside of the available functions : DEVICE MOUNTPOINT SIZE /dev/sda (total) 500,000 MB /dev/sda1 /boot/ 128 MB /dev/sda2 / 20,000 MB /dev/sda3 /var/log/ 20,000 MB /dev/sda5 /home/ 400,000 MB /dev/sda6 /opt/ 51,680 MB /dev/sda7 swap 8,192 MB</code></pre> Great, no problem - we can easily define that in the kickstart : part /boot --asprimary --size=128 part / --asprimary --size=20000 part /var/log --asprimary --size=20000 part /home --size=400000 part /opt --size=51680 part swap --size=8192</code></pre> But what happens if we want to use this same kickstart on another machine (or, indeed, many other machines) that don't have the same disk size ? One of the options that can be used with the « part » keyword is « --grow », which tells Anaconda to create as large a partition as possible. This can be used along with « --maxsize= », which does exactly what you think it does. Continuing with the example, we can modify the « /home » partition to be of a variable size, which should do us nicely on disks which may be smaller or larger than our original 500GB unit. part /home --size=1024 --grow</code></pre> Here we've stated that we'd like the partition to be at least a gig, but that it should otherwise be as large as possible given the constraints of both the other partitions, as well as the total space available on the device. But what if you also want « /opt » to be variable in size ? One way would be to grow both of them : part /home --size=1024 --grow part /opt --size=1024 --grow</code></pre> Now, what do you think that will do ? If you guessed « grow both of them to half the total available size each », you'd be correct. Maybe this is what you wanted - but then again, maybe it wasn't. Of course, we could always specify a maximum ceiling on how far /opt will grow : part /opt --size=1024 --maxsize=200000 --grow</code></pre> That works, but only at the potential expense of /home. Consider what would happen if this was run against a 250GB disk ; the other (static) partitions would eat up some 48GB, /opt would grow to the maximum specified size of 200GB, and /home would be left with the remaining 2GB of available space. If we were to add more partitions into the mix, the whole thing would become an imprecise mess rather quickly. Furthermore, we haven't even begun to look at scenarios where there may (or may not) more than one disk, nor any fun tricks like automatically setting the swap size to be same as the actual amount of RAM (for example). For these sorts of things we need a different approach. the magic of pre, the power of parted</h3> The kickstart configuration contains a section called « %pre</a> », which should be familiar to anybody who's dealt with RPM packaging. Basically, the pre section contains text which will be parsed by the shell during the installation process - in other words, you can write a shell script here. Fairly be thee warned, however, as the shell spawned by Anaconda is « BusyBox</a> », not « bash</a> », and it lacks some of the functionality that you might expect. We can use the %pre section to our advantage in many ways - including partitioning. Instead of using the built-in functions to set up the partitions, we can do it ourselves (in a manner of speaking) using « parted</a> ». Parted is, as you might expect, a tool for editing partition data. Generally speaking it's an interactive tool, but one of the nifty features is the « scripted mode », wherein partitioning commands can be passed to Parted on the command-line and executed immediately without further intervention. This is very handy in any sort of automated scenario, including during a kickstart. We can use Parted to lay the groundwork for the basic example above, wherein /home is dynamically sized. Initially this will appear inefficient, since we won't be doing anything that can't be accomplished by using the existing Kickstart functionality, but it provides an excellent base from which to do more interesting things. What follows (until otherwise noted) are text blocks that can be inserted directly into the %pre section of the kickstart config : # clear the MBR and partition table dd if=/dev/zero of=/dev/sda bs=512 count=1 parted -s /dev/sda mklabel msdos</code></pre> This ensures that the disk is clean, so that we don't run into any existing partition data that might cause trouble. The « dd » command overwrites the first bit of the disk, so that any basic partition information is destroyed, then Parted is used to create a new disk label. TOTAL=`parted -s /dev/sda unit mb print free | grep Free | awk '{print $3}' | cut -d "M" -f1`</code></pre> That little line gives us the total size of the disk, and assigns to a variable named « TOTAL ». There are other ways to obtain this value, but in keeping with the spirit of using Parted to solve our problems, this works. In this instance, « awk</a> » and « cut</a> » are used to extract the string we're interested in. Continuing on... # calculate start points let SWAP_START=$TOTAL-8192 let OPT_START=$SWAP_START-51680</code></pre> Here we determine the starting position for the swap and /opt partitions. Since we know the total size, we can subtract 8GB from it, and that gives us where the swap partition starts. Likewise, we can calculate the starting position of /opt based on the start point of swap (and so forth, were there other partitions to calculate). # partitions IN ORDER parted -s /dev/sda mkpart primary ext3 0 128 parted -s /dev/sda mkpart primary ext3 128 20128 parted -s /dev/sda mkpart primary ext3 20128 40256 parted -s /dev/sda mkpart extended 40256 $TOTAL parted -s /dev/sda mkpart logical ext3 40256 $OPT_START parted -s /dev/sda mkpart logical ext3 $OPT_START $SWAP_START parted -s /dev/sda mkpart logical $SWAP_START $TOTAL</code></pre> The variables we populated above are used here in order to create the partitions on the disk. The syntax is very simple : « parted -s » : run Parted in scripted (non-interactive) mode.</li> « /dev/sda » : the device (later, we'll see how to determine this dynamically).</li> « mkpart » : the action to take (make partition).</li> « primary | extended | logical » : the type of partition</a>.</li> « ext3 » : the type of filesystem (there are a number of possible options, but ext3 is pretty standard). Notice that the « extended » and « swap » definitions do not contain a filesystem type - it is not necessary.</li> </ul> </li> « start# end# » : the start and end points, expressed in MB.</li> </ul> Finally, we must still declare the partitions in the usual way. Take note that this does not occur in the %pre section - this goes in the normal portion of the configuration for defining partitions : part /boot --onpart=/dev/sda1 part / --onpart=/dev/sda2 part /var/log --onpart=/dev/sda3 part /home --onpart=/dev/sda5 part /opt --onpart=/dev/sda6 part swap --onpart=/dev/sda7</code></pre> As i mentioned when we began this section, yes, this is (so far) a remarkably inefficient way to set this particular basic configuration up. But, again to re-iterate, this exercise is about putting the groundwork in place for much more interesting applications of the technique. mo' drives, mo' better</h3> Perhaps some of your machines have more than one drive, and some don't. These sorts of things can be determined, and then reacted upon dynamically using the described technique. Back to the %pre section : # Determine number of drives (one or two in this case) set $(list-harddrives) let numd=$#/2 d1=$1 d2=$3</code></pre> In this case, we're using a built-in function called « list-harddrives » to help us determine which drive or drives are present, and then assign their device identifiers to variables. In other words, if you have an « sda » and an « sdb », those identifiers will be assigned to « $d1 » and « $d2 », and if you just have an sda, then $d2 will be empty. This gives us some interesting new options ; for example, if we wanted to put /home on to the second drive, we could write up some simple logic to make that happen : # if $d2 has a value, it's that of the second device. if [ ! -z $d2 ] then HOMEDEVICE=$d2 else HOMEDEVICE=$d1 fi # snip... part /home --size=1024 --ondisk=/dev/$HOMEDEVICE --grow</code></pre> That, of course, assumes that the other partitions are defined, and that /home is the only entity which should be grown dynamically - but you get the idea. There's nothing stopping us from writing a normal shell script that could determine the number of drives, their total size, and where the partition start points should be based on that information. In fact, let's examine this idea a little further. the size, she is dynamic !</h3> Instead of trying to wrangle the partition sizes together with the default options, we can get as complex (or as simple) as we like with a few if statements, and some basic maths. Thinking about our layout then, we can express something like the following quite easily : If there is one drive that is at least 500 GB in size, then /opt should be 200 GB, and /home should consume the rest.</li> If there is one drive is less than 500 GB, but more than 250 GB, then /opt and /home should each take half.</li> If there is one drive that is less than 250 GB, then /home should take two-thirds, and /opt gets the rest.</li> </ul> # $TOTAL from above... if [ $TOTAL -ge 512000 ] then let OPT_START=$SWAP_START-204800 elif [ $TOTAL -lt 512000 ] && [ $TOTAL -ge 256000 ] then # get the dynamic space total, which is between where /var/log ends, and swap begins let DYN_TOTAL=$SWAP_START-40256 let OPT_START=$DYN_TOTAL/2 elif [ $TOTAL -lt 256000 ] then let DYN_TOTAL=$SWAP_START-40256 let OPT_START=$DYN_TOTAL/3 let OPT_START=$OPT_START+$OPT_START fi</code></pre> Now, instead of having to create three different kickstart files, each describing a different scenario, we've covered it with one - nice ! other possibilities</h3> At the end of the day, the possilibities are nearly endless, with the only restriction being that whatever you'd like to do has to be do-able in BusyBox - which, at this level, provides a lot great functionality. Stay tuned for more entries related to kickstarting, PXE-based installations, and so forth, all to come here on dan's linux blog. Cheers ! getting Dia to give you a pdf 2009-07-16T00:00:00+00:00 Hello again ! Today's quick tip concerns a software package called Dia</a>, which is an open source tool (available for both Windows and Linux, as it goes) used to make diagrams, flowcharts, network maps, and so forth. It has its own file format (.dia), which is (obviously?) useful for saving the projects you're working on, but less useful if you need to give the diagram to anybody else, either in print or electronic form. Dia can export to a variety of formats including SVG</a>, PNG</a>, and EPS</a>, but one export format that it lacks native support for is the venerable PDF, which has become a de facto standard for transmitting documents between diverse environments. There are many advantages and interesting aspects of the PDF format, not the least of which being that what you see on your screen is what you get when it's printed. It is unfortunate, then, that Dia won't spit out a PDF (even if you ask very nicely). Of course, being that it's so easy to print directly to PDF (via CUPS</a>, for example) these days, having native support for PDF may not, at first, seem all that useful. Well, as it turns out, printing directly to PDF might not give you quite what you were looking for. In practice, you do get a PDF, but what appeared to be a modestly-sized diagram in Dia will turn out to be a multi-page monster in (virtually) printed form. As a general rule, this is not what you want. In order to get a usable PDF we need to use an intermediate step between Dia and the final file. The idea, quite simply, is to export the diagram as one of the supported formats, then convert that file into a PDF. There are a number of options here, but for our purposes we'll save the diagram as an EPS file, then use a quick little command-line tool called « epstopdf » to perform the conversion. There's a good chance that you don't have epstopdf on your machine. If you're using Ubuntu, you used to be able to install it easily via the APT packager, but these days the little conversion tool comes as part of a larger suite of tools called « texlive-extra-utils ». This suite is dependant on a number of other packages, so go ahead and install them all : $ sudo apt-get install texlive-extra-tools</code></pre> EDIT : In Ubuntu 10.04, the package is named « texlive-font-utils ». Among many, many little items of interest, our target application will be installed. To use it, simply feed it the name of the EPS file as an argument : $ epstopdf somediagram.eps</code></pre> It will automatically output a PDF file of the same name. There you go - a nice, shiny PDF of your Dia diagram. Enjoy ! pohmelfs pt. 2, return of pohmelfs ! 2009-07-06T00:00:00+00:00 Hello again fair readers. Today i'm going to re-visit POHMELFS, which i introduced in an earlier blog post</a>. I received a comment on that post which basically asked for more information on some of the more interesting (read : advanced) features of POHMELFS, such as distributed storage, and the like. Well, today is the day ! If you need a refresher, be sure to skim over my previous post, as we're going to dive in now right where i left off last time. patch for the win</h3> One of the reasons that there was a bit of a delay between my last POHMELFS post and this one was because i hit a bug. Given that we're working with staging-level code here, that's to be expected - luckily, thanks to some quick work by Evgeniy Polyakov on the POHMLEFS mailing list</a>, there is still hope - hope in the form of a tasty little patch. diff --git a/drivers/staging/pohmelfs/trans.c b/drivers/staging/pohmelfs/trans.c index eab7868..bf7b09a 100644 --- a/drivers/staging/pohmelfs/trans.c +++ b/drivers/staging/pohmelfs/trans.c @@ -467,7 +467,8 @@ int netfs_trans_finish_send(struct netfs_trans *t, struct pohmelfs_sb *psb) continue; } - if (psb->active_state && (psb->active_state->state.ctl.prio >= st->ctl.prio)) + if (psb->active_state && (psb->active_state->state.ctl.prio >= st->ctl.prio) && + (t->flags & NETFS_TRANS_SINGLE_DST)) st = &psb->active_state->state; err = netfs_trans_push(t, st);</code></pre> Basically, this patch fixes a minor, but ultimately crippling bug related to writing to multiple servers. The details are not important - what's important is that we apply the patch and keep the dream alive. First, you'll need to copy and paste that block of code into a text file on one of the systems (in « ~/pohmel.diff, for example »). Then, in order to apply the patch, we'll need to use a standard tool called (appropriately) « patch » : [root@host_75 ~]# cd /usr/src/linux [root@host_75 ~]# patch -p 1 < ~/pohmel.diff patching file drivers/staging/pohmelfs/trans.c</code></pre> Now, just as we did last time, we must play the kernel and module compilation and installation game (fun!). If you need a refresher on how to do this, just go back to my previous post</a>. Note that this time around, the whole process will be much faster, since only the POHMELFS components need to be recompiled - everything else will stay the same. As a result, you can skip the part where you archive the entire kernel tree and copy it over - instead, just patch and recompile on each server and the client. It's your call. Once that's out of the way we'll reboot, and then it's off to the races. a new challenger appears !</h3> It's now time to add a third machine into the mix (« host_147 » in this case). Using this new box, we'll create a simple sort of setup which is, in fact, quite representative of how things might work in the real world : two storage servers and a client. As you no doubt recall, one of the neat features of POHMELFS is that it can be employed in a parallel fashion, meaning that a file which appears to the client to be in one place, is actaully located in more than one storage medium. A general way of describing these ideas is by using the terms « logical » and « physical » ; the logical medium is the filesystem that the client sees, and the physical medium is the actual hard drive upon which the data is stored. In this case, host_75 and host_166 will be the servers, each containing one copy of the data on their respective physical mediums (i.e. hard drives), and host_147 will be our client, which will access the data via the logical medium (i.e. the POHMELFS export). The new machine was set up in the same way as host_166 was, so we'll skip over that, and get right to the good stuff. A new directory should be created on each of the machines : « /opt/pohtest ». This will serve as the export directory on the servers, and the mount directory on the client - don't put any data in it yet, though. server config</h3> On the servers, we'll initiate the server daemon. Unlike our first test, where we just let the defaults ride, this time around we'll configure things a bit more intelligently : [root@host_75 (and host_166) ~]# fserver -r /opt/pohtest -l /var/log/pohmelfs.log -d</code></pre> In the above example, « -r » defines the directory to export, « -l » is where to output the logs to, and « -d » puts the process into the background, instead of on our console as before. This is normally how things would work, so it's good to get used to it now. Now, we can follow the log files on each machine by using « tail » : [root@host_75 (and host_166) ~]# tail -f /var/log/pohmelfs.log Server is now listening at 0.0.0.0:1025.</code></pre>client config</h3> With the servers up and ready to go, we can now turn our attention on the client. Don't forget to load the pohmelfs module first ! [root@host_147 ~]# modprobe pohmelfs [root@host_147 ~]# cfg -A add -a 192.168.0.75 -p 1025 -i 1</code></pre> Now we mount. It's important that we mount before we attempt to add the second server into the mix - trying to do it ahead of time will only result in terrible, crippling failure. [root@host_147 ~]# mount -t pohmel -o idx=1 none /opt/pohtest/</code></pre> No output means it worked (as usual), so let's verify : [root@host_147 ~]# df | grep poh none 154590376 10018492 144571884 7% /opt/pohtest</code></pre> Great, now let's add the other server : [root@host_147 opt]# cfg -A add -a 192.168.0.166 -p 1025 -i 1</code></pre> Now we must wait at least 5 seconds for the synchronisation to occur. In reality it's shorter than that, but 5 seconds is an easy number to remember, and it's safe. So far this looks exactly the same as before, but there's a bit of a conceptual twist - as you can see, both of those new add statements have the same index (as denoted by the -i). This means that they're grouped together as part of the same logical medium. We can check on this by using the « show » action : [root@host_147 ~]# cfg -A show -i 1 Config Index = 1 Family Server IP Port AF_INET 192.168.0.75 1025 AF_INET 192.168.0.166 1025</code></pre> Everything seems on the up and up so far, so we can go ahead and try our first mount. A series of options will be passed to the mount line, notably « idx=1 », which means index 1 (as seen above) - this is very important to specify, as without it, POHMELFS won't be able to determine which logical group you're talking about. And if we take a look at the log output on the servers, we'll see that the client connection has been accepted. Both of the logs should show the accepted line, but with different port numbers (the trailing digits at the end) : Accepted client 192.168.0.147:48277.</code></pre> There are other diagnostics we can run to take a look at what we've got running. At this stage they won't tell us anything we don't already know, but it will give us some practice with the tools and data, so that when the time comes to debug problems down the road, we'll be ready. For example, POHMELFS will write some handy information to « mountstats », which is exactly what it sounds like : [root@host_147 ~]# cat /proc/1/mountstats ... device none mounted on /opt/pohtest with fstype pohmel idx addr(:port) socket_type protocol active priority permissions 1 192.168.0.75:1025 1 6 1 0 3 1 192.168.0.166:1025 1 6 1 0 3</code></pre> It's not lined up very nicely, but the interesting column right now is « active », which lists « 1 » in both cases, meaning the connections are open. The « permissions » column lists « 3 » for both nodes which, in this case, means that they're both available for reading and writing (as opposed to being read or write-only, which are also valid options). but will it blend ?</h3> Accepting the connection is one thing - successfully reading and writing files is entirely another. Let's do some tests ; first we'll use the client to create an empty file in mount : [root@host_147 ~]# cd /opt/pohtest/ [root@host_147 pohtest]# touch FILE [root@host_147 pohtest]# ls FILE</code></pre> Great, now let's take a look at our servers : [root@host_166 pohtest]# ls -l total 0 -rw-r--r-- 1 root root 0 2009-07-06 16:58 FILE [root@host_166 ~]#</code></pre> And the other : [root@host_75 ~]# ls -l /opt/pohtest/ total 0 -rw-r--r-- 1 root root 0 2009-07-06 16:46 FILE [root@host_75 ~]#</code></pre> Now, during my limited tests, i noticed a small lag time between my manipulations on the client, and when those actions were reflected on the servers. At this stage of the game i'm not sure whether that's normal or not, or exactly what's causing it - so don't be alarmed if you see a small lag as well. I'll be sure to post further updates on this point once i've got more information. Update : As per Evgeniy on the mailing list : This delay is not a bug, but feature - POHMELFS has local cache on clients and data written on client is stored in that cache first and then flushed to the server when client is under memory pressure or when another one requests updated but not yet flushed data. To force client to flush the data one can 'sync' on client or use 'flush' utility on the server. The latter will invalidate data on the client (which implies it to be flushed to the server first), so server update will become visible next time client reads that data.</code></pre>how not to do it</h3> Let's do another little test. On one of the servers, we'll perform a manipulation in the POHMELFS export directory : [root@host_75 ~]# touch /opt/pohtest/host75file [root@host_75 ~]# ls -l /opt/pohtest/ total 4 -rw-r--r-- 1 root root 5 2009-07-06 16:46 FILE -rw-r--r-- 1 root root 0 2009-07-06 16:57 host75file [root@host_75 ~]#</code></pre> Great, but if we take a look at the other server : [root@host_166 ~]# ls -l /opt/pohtest/ total 4 -rw-r--r-- 1 root root 5 2009-07-06 16:59 FILE</code></pre> And the client : [root@host_147 ~]# ls -l /opt/pohtest/ total 0 -rw-r--r-- 1 root root 5 2009-07-06 20:47 FILE</code></pre> We notice that it's not there. Why ? Unfortunately, like so much bureaucracy, we didn't go through the proper channels. Recall that our client has certain software running on it that allows it to speak to both servers, and that the mountpoint uses that software to ensure consistency between across the shared filesystem. In the example above, we wrote directly to the underlying filesystem of the server - completely avoiding said software - and thus POHMELFS had no way of knowing that a manipulation had occured. In short - if you want to keep things consistent, you must interact via a client. But what if we want our servers to be able to interact with the data as well ? Well, there's nothing stopping us from setting up client processes on our servers, too. This, however, will have to wait for the next instalment. See you on the intertubes ! setting the from address in GNU mail 2009-07-03T00:00:00+00:00 Today i'm going to address a question that comes up again and again on the various Linux help forums and mailing lists (in one form or another) : « How do i set the From address when using mail ? » Of course, « mail » in this case refers to that most basic of all Linux mail applications : GNU mail. Commonly part of the default software on a Linux box, it's not used very often in an interactive capacity, but it still gets a lot of play in the System Administration world as a way to quickly fire off automated emails from scripts. For example : $ echo "This is the message" | mail -s "This is the subject" user@domain.tld</code></pre> This would result in a message that's from whatever raw user@system that the local mailing software detects, such as « systemcheck@2.1.0.192.int.domain.tld » or some such thing. Normally, whatever defaults (including the From address) that get used are good enough, but occasionally it can be handy to have a finer level of control ; for example, policy restrictions on your local mail relay, or just because the default From address looks bad and you want to make it more readable. The solution, like so many others in the Linux world, is straightforward once you already know about it, but just obscure enough that it isn't obvious at first. The man page for « mail » on Ubuntu (and, likely, other distros as well) has this tasty little morsel of information listed as the very first option : -a, --append=HEADER: VALUE Append given header to the message being sent</code></pre> « Headers » form part of every email message, and are used to store all sorts of information about the email, from such pedestrian items as « From: » and « To: », to more esoteric things like Spam analysis breakdowns and binary encoding methods. For now, the important thing to realise is that the From: address is contained within the From header, and using the argument noted above, we can set it quite easily : $ echo "The message" | mail -s "The subject" --append=FROM:sensible@domain.tld user@domain.tld</code></pre> This would send the same message as above, but with the From: address set to « sensible@domain.tld ». In fact, any header can be altered in this fashion, not just the From header - so if you ever have the need to change other headers, now you know at least one way. :) pohmelfs update 2009-07-01T00:00:00+00:00 Hello again ! You may be wondering when the next update in the POHMELFS series is coming - well, rest assured that i'm working on it even as you read this, and that it will be worth the wait. Remember that we're working with Staging-level code, and that sometimes things don't always go as well as one might hope - in this case, there are some discrepancies between the code that the POHMELFS devs are using, and that which was released in the 2.6.30 code (according to the devs, at least). I've recently received a nice new patch from one of the devs, and once i've got that squared away, we'll continue with our exercise. force disk geometry with sfdisk 2009-06-22T00:00:00+00:00 Hello again ! This is a quick and dirty update which covers a handy little trick when dealing with writeable removable media - especially USB drives, compact flash cards, and the like. I end up using a lot of USB keys in my environment for a variety of reasons, not the least of which is as handy portable Linux drives that can be stuck into any workstation and booted from directly. They're like LiveCDs, except since i can write to them, any changes that are made during a session don't disappear when the machine reboots (nice). As an aside, if that sounds interesting to you, i suggest checking out the Fedora LiveCD on USB Howto</a>. USB keys are so ubiquitous now that we buy in bulk, meaning we'll get a bunch of identical units at one time. Once in a while (though more often than i'd like), one of the keys will end up having a detected geometry which is different from the others. This isn't normally a big deal, but it can cause slight variations in the apparent available space to create a partition. This ends up being a problem if i'm looking to clone data from one key to another using a disk imaging tool such as « Partimage</a> » (another tool that gets a lot of play around here). The solution is fantastically simple, but perhaps not immediately obvious, as it requires the use of a tool that - for the most part - never gets touched by the average user (or admin !) : « sfdisk</a> ». Sfdisk is a partition table manipulator that allows us to do a number of advanced (read: dangerous) operations to disks. Since the common day-to-day operations one might perform on a disk, such as creating or modifying partition assignments, are covered by the more common « fdisk</a> » (or even « cfdisk</a> »), sfdisk is rarely called upon outside of bizarre or extreme situations. Altering geometry is one such situation. change is good</h3> The first thing we need to do is determine what the correct geometry is. This is obtained easily enough by running an fdisk report against a known-good key (sdc, in this case) : [root@host_166 ~]# fdisk -l /dev/sdc Disk /dev/sdc: 4001 MB, 4001366016 bytes 19 heads, 19 sectors/track, 21648 cylinders Units = cylinders of 361 * 512 = 184832 bytes Disk identifier: 0xf1bcd225 Device Boot Start End Blocks Id System /dev/sdc1 * 1 21648 3907454+ 83 Linux</code></pre> Alternatively, we could ask sfdisk : [root@host_166 ~]# sfdisk -g /dev/sdc /dev/sdc: 21648 cylinders, 19 heads, 19 sectors/track</code></pre> Now that we have the correct geometry, we can get sfdisk to alter that of the naughty key (sdb, in this case). As you can likely guess, -C is the cylinders, -H is the heads, and -S in the sectors (per track) : [root@host_166 ~]# sfdisk -C 21648 -H 19 -S 19 /dev/sdb</code></pre> Depending on your particular version of sfdisk and distro, this may trigger an interactive process which will ask you to create the desired partitions on the key. Assuming you just want one big Linux partition, you can hit « enter » and accept every default until it's done. And that's that - one key brought rapidly in line with the others. Cheers ! pohmelfs - the network filesystem of the future ! 2009-06-17T00:00:00+00:00 Hello again fair readers ! Today we're going to take a look at POHMELFS</a>, which is a network file system that was just recently integrated into the Linux kernel. This is an excellent exercise for three reasons : we'll learn about some great new concepts related to network file systems, we get to compile and install our own kernel, and we get to play with a great new platform that could, eventually, become the new standard for network file systems in the *nix world. Sound good ? Let's go ! A network file system is, according to Wikipedia</a>, « any computer file system that supports sharing of files, printers and other resources as persistent storage over a computer network. » It is important to note that there is also a specific protocol called « Network File System », which is exactly what it sounds like, and is highly popular in the *nix world. For the purposes of this article, unless otherwise stated every time i use « network file system » or « nfs », i'm referring to the concept, not the protocol. Building on the idea of an nfs, there are other types of network-aware file systems which fall into similar categories, such as « distributed file systems</a> », « parallel file systems », « distributed parallel file systems », and so forth. These terms are largely non-standard, and conventional wisdom tends to put all of these sub-categories into the one big nfs family. POHMELFS, for example, is described by its creator as a « distributed parallel internet filesystem » - in fact, the name itself is an acronym for « Parallel Optimized Host Message Exchange Layered File System », which is, mercifully, pronounced simply as « poh-mel ». The two interesting portions of the name are « parallel » and «optmized »... parallel</h3> One of the interesting aspects of POHMELFS is that a given chunk of data can (and in many cases, should) reside in more than one distinct storage unit (different physical servers, for example). The client can be made aware of this, which means that accesses to the data can happen in a « parallel » fashion, which has two major advantages : Seamless real-time replication of the data during write operations</li> Faster overall data accesses during read operations</li> </ul> Imagine a scenario whereby you have two file servers : serverA and serverB, and any given number of clients (clientA, B, C, etc..). In a classic non-parallel (and non-replicating) file system scenario, data would not be identical between the two file servers. Clients would need to know which server had the data they wanted ahead of time, and if one of the servers went down, it would take all of its unique data with it. Furthermore, if all of the clients wanted to read data from serverB at once, everybody would get bogged down overall, since there are a finite amount of resources available for everybody to use. The same scenario with a parallel file system is much better indeed ! Firstly, data would be identical on the two file servers, meaning that if one were to go down, there would be no direct loss of data availability (already a huge improvement). Secondly, and this is also huge, clients could spread their requests for data between the two servers, thus reducing the direct load on any one given server, and resulting in better overall resource usage (translation : better performance). optimized</h3> The author claims that POHMELFS is designed from the ground-up with performance in mind. This design principle has resulted in some amazing benchmarks which, frankly stated, make more or less every other network file system look slow</a> in comparison</a> ; some types of data access processes are merely rapid, whereas others are revolutionary. Of course, speed isn't everything, and as you might expect, POHMELFS isn't quite as feature rich (yet?) as many of the other nfs options on the market. take it for a test drive</h2> At the top of the article i mentioned that POHMELFS was recently introduced into the Linux kernel. What i really meant was that as of kernel version 2.6.30, POHMELFS is located in the « staging</a> » tree of the overall source</a> collection. This is already a little bit of a warning - code in the staging area is generally considered usable, but not ready to be merged into the kernel proper. If this doesn't sound like your bag, it's time to bail out now, but for those of you in the mood for adventure, the staging tree is a great place to find interesting new features and functionalities. In the test scenario we're about to run through, there are two servers : « host_75 » and « host _166 ». Both of these machines are very basic installations of Fedora 8, which from a server perspective is not very different from any other Fedora prior or since - therefore, unless otherwise noted, these operations should be identical on any other Fedora box. kernel configuration</h3> Long ago, in a faraway land where dragons battled wizards for supremacy over ancient battlements, the process of properly configuring, compiling, and installing a new Linux kernel was arcane knowledge - the stuff of legends. In our modern, enlightened age, installing a new kernel on your machine is (relatively) simple ! Heck, you even get a menu these days... The first step in compiling a new kernel is to make sure that your system has the necessary tools with which to work. On a standard Fedora system, the fastest way is simply to install all of the development tools as a single mass operation. This is overkill, really, but it's simple, and since we're just testing things out anyways, fast and simple are our primary criteria : [root@host_75 ~]# yum groupinstall 'Development Tools'</code></pre> The kernel menu i mentioned a few moments ago is going to require an additional package which is not part of the development tools group : « ncurses-devel ». [root@host_75 ~]# yum install ncurses-devel</code></pre> Next up, we need to download and unpack the kernel source package. Your best bet is from kernel.org</a>, which is reliable, rapid, and most importantly, secure : [root@host_75 ~]# cd /usr/src [root@host_75 src]# wget http://eu.kernel.org/pub/linux/kernel/v2.6/linux-2.6.30.tar.bz2 [root@host_75 src]# tar -xvjf linux-2.6.30.tar.bz2 [root@host_75 src]# ln -s linux-2.6.30 linux</code></pre> Finally, we can take a look at the configuration menu. You'll notice that we issue the command « make », which is a mechanism for compiling source code used across the *nix world. We'll see it again a little later on, but for now, understand that all we're doing here is «making » the configuration menu, not the kernel itself. [root@host_75 src]# cd linux [root@host_75 linux]# make menuconfig</code></pre> Now there are a lot (a LOT) of possible options for configuring a kernel, and if you've got the time and the inclination, going through each one and reading the description can be very enlightening. That said, what we're interested in is POHMELFS, and in order to enable it, we're going to have to explicitly tell the configuration that we're interested in staging-level code. First, enable « Staging Drivers » : Device Drivers ---> [*] Staging Drivers</code></pre> Then disable « Exclude Staging drivers from being built ». It's set up this way in order to prevent somebody from accidentally building anything from staging by accident : Device Drivers ---> Staging Drivers ---> [ ] Exclude Staging drivers from being built</code></pre> Next, enable POHMELFS as a module (if you'd like a refresher on modules, just check out any post on this blog with the « modules » tag) : Device Drivers ---> Staging Drivers ---> <M> POHMELFS filesystem support</code></pre> And, optionally, support for encryption : Device Drivers ---> Staging Drivers ---> <M> POHMELFS filesystem support > [*] POHMELFS crypto support</code></pre> Finally, you may wish to add a « local version string » - this is an identifier that you can customise to help you keep track of each kernel build. General Setup ---> (-pohmelfs_test) Local version</code></pre> Now we save and exit ! build & install the kernel</h3> From here, all that's left is to let it build - depending on your hardware, this can take a while. Patience, grasshopper. [root@host_75 linux]# make && make modules && make modules_install && make install</code></pre> There are four distinct commands which, assuming the previous one is successful, will execute consecutively - that's what the double-ampersand (&&) does. make : Builds the actual kernel (this is the part that takes forever)</li> make modules : Builds the modules (everything enabled as , such as POHMELFS)</li> make modules_install : Copies the modules to their proper positions</li> make install : Creates the initrd (which i discussed in a previous post), sets up the bootloader (which we'll take a look at), and so forth</li> </ul> Once the process is done, which is to say that all four items executed successfully, the last thing we need to check before we reboot is the bootloader - in this case, « GRUB</a> ». The « make install » will add an entry for our new kernel into the GRUB configuration. This is fairly automatic, but i like to check it, just to be safe. The new entry should look something like this : title Fedora (2.6.30-pohmelfs_test) root (hd0,0) kernel /vmlinuz-2.6.30-pohmelfs_test ro root=/dev/sda1 initrd /initrd-2.6.30-pohmelfs_test.img</code></pre> From here, we reboot, and when the GRUB menu appears, select our new POHMELFS item instead of the default entry. userspace tools</h3> The actual POHMELFS executables - the « userspace tools », so called since they are used by the user, not by the system, are not included with the kernel. This is normal. Even though our kernel now supports POHMELFS, our system doesn't actually have any of the software which will interface with the kernel module yet. This has to be downloaded, configured, and compiled in the same fashion as the kernel ; however, whereas the kernel was easily downloaded via HTTP, the POHMELFS source is only available via « GIT</a> ». GIT, in a nutshell, is a platform for managing source code (like « CVS</a> », « Subversion</a> », or « VSS</a> », just to name a few). For our purposes today, we're only going to use one of its many features : copying the source code from the official POHMELFS site so that we can compile it ourselves. If you don't already have GIT installed, now is the time ! Don't delay, act today ! [root@host_75 ~]# yum install git</code></pre> Depending on your existing installation, this may cause a fairly large number of new packages to be downloaded and installed, so don't worry if the list looks huge. With that out of the way, we can download the source - this is known as « cloning » a « project » in GIT terminology : [root@host_75 ~]# git clone http://www.ioremap.net/git/pohmelfs-server.git</code></pre> Preparation of the source and so forth for POHMELFS was, not too long ago, a bit of a pain in the yoohoo. Now, the author has graciously included a tool which will take most of the pain out of the process - though there are still a few things to look out for : [root@host_75 ~]# cd pohmelfs-server [root@host_75 pohmelfs-server]# ./autogen.sh</code></pre> This will take a little while, and will output a handful of lines as it goes along. The next step is to is a standard « ./configure », which, if you've never compiled anything before, is just about the most standard possible way to pre-configure source for compilation in the *nix world. Normally, ./configure accepts a variety of options (take a look at « ./configure --help » for a taste), but for now, we're only interested in one : [root@host_75 pohmelfs-server]# ./configure --with-kdir-path=/usr/src/linux/drivers/staging/pohmelfs</code></pre> The supplied option tells ./configure where the POHMELFS kernel code is - in specific, where the « netfs.h » file is located. This is important later on, so take note. This process output tonnes of lines as it checks for the capabilities and desires of your environment, and customises the configuration as best it can for your machine. Once it's done, we can go ahead and « make » : [root@host_75 pohmelfs-server]# make</code></pre> As of this writing, the above make may fail. As it turns out, certain elements in the POHMELFS source, as they stand now, expect OpenSSL</a> to be installed on the system. This is true even if you were clever and provided « --disable-openssl » to ./configure above (good thinking, though !). We've got two options here : either modify the POHMELFS source in order to remove the references to things which do not exist, or simply install OpenSSL and be done with it. If you've already got OpenSSL on your system, then no worries, you probably didn't even see this problem. OpenSSL, briefly, is an open-source implementation of a series of encryption protocols and cryptographic algorithms which, among other things, allow for « secure » websites (i.e. via HTTPS), and other such things. As such, it's a pretty standard thing to have on a machine (especially a server), and since it's so easy to install in our test scenario, we'll just go ahead and do that now : [root@host_75 pohmelfs-server]# yum install openssl openssl-devel</code></pre> Back to POHMELFS, it's time to reconfigure. We'll enable openssl now, since we've got it anyways... [root@host_75 pohmelfs-server]# ./configure --with-kdir-path=/usr/src/linux/drivers/staging/pohmelfs --enable-openssl [root@host_75 pohmelfs-server]# make</code></pre> As of this writing, the above make will fail (again, possibly). In this instance, a required file can't be located : netfs.h . Remember that one from above ? Of course you do. As it turns out, even though we explicitly specified the path to find this file, certain elements in the source (possibly auto-generated) expect it to be elsewhere. As with OpenSSL, we have two options : alter the code ourselves, or just satisfy the requirement as painlessly as possible. Well, you already know how we roll in these parts : [root@host_75 pohmelfs-server]# mkdir /usr/src/linux/drivers/staging/pohmelfs/fs [root@host_75 pohmelfs-server]# ln -s /usr/src/linux/drivers/staging/pohmelfs /usr/src/linux/drivers/staging/pohmelfs/fs/pohmelfs</code></pre> What we've done here is create a link that points from where the source wants netfs.h to be, to where it actually is. Hey, it's staging-level code, remember ? No worries - this was an easy one anyways. With that out of the way, we can make away ! [root@host_75 pohmelfs-server]# make ... [root@host_75 pohmelfs-server]# make install</code></pre> The make install will put the necessary binaries in the appropriate places on the system. In particular : [root@host_75 ~]# which fserver cfg /usr/local/bin/fserver /usr/local/bin/cfg</code></pre>and again !</h3> That's one server down, one to go. But, wait, that was a lot of work, and even more waiting, wasn't it ? Doing it again would suck ; luckily, there are some shortcuts we can take. If the hardware of both machines is more or less the same, there's a better-than-average chance that the same kernel you've already compiled will work on the other server - you can just port it over. Now, there are very particular, very clean ways to go about this, and to those that like their test environments clean and tidy, i salute you ; we, however, know better. Let's just pack up the source, copy it over, and deploy it all at once : [root@host_75 ~] cd /usr/src/ [root@host_75 src] tar -cvzf src.tar.gz linux ... [root@host_75 src] scp src.tar.gz root@host_166:/usr/src/ ...</code></pre> From here on in you'll want to keep an eye on the hostname being used - we're dealing with two machines now... [root@host_166 ~] cd /usr/src/ [root@host_166 src] tar -xvzf src.tar.gz [root@host_166 src] cd linux [root@host_166 src] make modules_install && make install</code></pre> Nice ! Reboot the second machine now, and don't forget to choose the new kernel from the GRUB menu. Likewise, we don't need to install GIT on the second machine - we'll just do like we did with the kernel : [root@host_75 ~]# tar -cvzf poh.tar.gz --exclude=.git pohmelfs-server/ [root@host_75 ~]# scp poh.tar.gz root@host_166:~/</code></pre> Notice the « --exclude » line in the tar command ? This is to prevent the GIT-specific stuff (which is substantial) from being archived, as it is not useful where we're going. [root@host_166 ~]# tar -xvzf poh.tar.gz ... [root@host_166 ~]# cd pohmelfs-server/ [root@host_166 pohmelfs-server]# make install</code></pre>testing time</h3> The first thing we need to do is load the POHMELFS module, which was generated way back when we built the kernel : [root@host_75 ~]# modprobe pohmelfs [root@host_75 pohmelfs-server]# lsmod Module Size Used by pohmelfs 59284 0</code></pre> You will likely have a lot more items in this list - but one of them must be « pohmelfs ». Before we start the server daemon, we'll have to decide which directory we want to « export », which is to say which one we'd like to share on the network. For now, let's pick « /tmp », since it's simple (and it's the daemon default). Let's put a file in there so that we can check to see if our share is properly working : [root@host_75 ~]# touch /tmp/POHTEST.TXT</code></pre> Next, we start the server daemon. For our first test, we'll just launch the binary without any options - by default, the daemon will launch in a local console, export /tmp (as noted above), and bind the process to port 1025. Eventually you may wish to change some or all of these defaults, but for now, we'll keep it simple : [root@host_75 ~]# fserver Server is now listening at 0.0.0.0:1025.</code></pre> The most basic test possible at this point is to telnet from the second machine to the first, just to see if we can connect on the port : [root@host_166 ~]# telnet host_75 1025 Trying 192.168.0.75... Connected to 192.168.0.75. Escape character is '^]'. ^] telnet> quit Connection closed.</code></pre> This will create some output on the server console : Accepted client 192.168.0.166:49744. fserver_recv_data: size: 40, err: -104: Success [0]. Dropped thread 1 for client 192.168.0.166:49744. Disconnected client 192.168.0.166:49744, operations served: 0. Dropping worker: 3086465936.</code></pre> Looks good ! Let's try a proper connection with the POHMELFS userspace tool : « cfg ». The options we'll pass to it are the most basic possible set : « -A add » : Action is to add a new connection</li> « -a » : Connect to this server</li> « -p » : Connect on this port</li> </ul> [root@host_166 ~]# cfg -A add -a 192.168.0.75 -p 1025 Timed out polling for ack main: err: -1.</code></pre> Uh oh ! What happened ? The error message tells us that the client « timed out » (or waited too long) to receive an acknowledgement of the connection from the server. But why ? The answer, though simple, is not immediately obvious : we forgot to load the POHMELFS module on the second machine. No worries, go ahead and do that now, and we'll try again : [root@host_166 ~]# modprobe pohmelfs [root@host_166 ~]# cfg -A add -a 192.168.0.75 -p 1025 [root@host_166 ~]#</code></pre> In a stroke of user-friendliness to last the ages, a successful cfg execution will produce no output. No news is good news, i suppose... Alright, now that we've got the server up, and we've prepared the client for a connection, we'll need to pick a spot to « mount » the remote share, then initiate the mount itself : [root@host_166 ~]# mkdir pohtest [root@host_166 ~]# mount -t pohmel -o idx=1 none pohtest/ mount: unknown filesystem type 'pohmel'</code></pre> Curses, foiled again ! Well, i was, at least - your mileage may vary on this one. If you get this error instead of a successful mount, the problem is very likely that the « pohmel » file system type isn't declared in all the proper places. Check « proc » and the « filesystems » etc config : [root@host_166 ~]# cat /proc/filesystems | grep poh nodev pohmel</code></pre>[root@host_166 ~]# cat /etc/filesystems | grep poh [root@host_166 ~]#</code></pre> Ah ha ! Let's rectify that little oversight and try again : [root@host_166 ~]# echo "nodev pohmel" >> /etc/filesystems [root@host_166 ~]# mount -t pohmel -o idx=1 none pohtest/</code></pre> No output ? Great success ! Let's verify : [root@host_166 ~]# df | grep poh none 154590376 10007348 144583028 7% /root/pohtest</code></pre> The server console also confirms the connection : fserver_root_capabilities: avail: 148053020672, used: 10247524352, export: 0, inodes: 39911424, flags: 2. Accepted client 192.168.0.166:37617.</code></pre> And, finally, we can see our test file : [root@host_166 ~]# cd pohtest [root@host_166 pohtest]# ls -l total 0 -rw-r--r-- 1 root root 0 2009-06-16 15:39 POHTEST.TXT</code></pre> Closing the connection cleanly is as simple as umounting : [root@host_166 ~]# umount pohtest</code></pre> This is confirmed on the server console : Dropped thread 1 for client 192.168.0.166:58955. Disconnected client 192.168.0.166:58955, operations served: 1. Dropping worker: 3086400400.</code></pre>that's a wrap, for now</h3> Now i know what you're thinking : where's the parallel storage ? Where's the real-time mirroring and all that fun stuff ? It's coming. For now, we're just getting our feet wet with technology. As time and testing permits, i'll post more about POHMELFS - so stay tuned ! UPDATE : Check out the next instalment in the series ! http://www.dark.ca/2009/07/06/pohmelfs-pt-2-return-of-pohmelfs/</a> Last but not least, be sure to check out « fserver -h » for such useful features as « fork to background » and « logfile » - both of which, i guaruntee, you'll want to look into if you intend to play around anymore with POHMELFS. Finally, remember that while the code is very mature for staging, it's still considered highly experimental. Good luck, and happy hacking ! how to be properly lazy, with perl ! 2009-06-12T00:00:00+00:00 One of the wonderful things about Perl is that it enables the busy System Administrator to be lazy - and that's a good thing ! Of course, i don't mean lazy as in unmotivated, or possesed of a poor work ethic, i mean it in the sense that Perl lets us do as little work as possible in a wide variety of situations. Let's examine this idea, shall we ? In the computer world, one often finds themselves doing the same sorts of things over and over again, such as adding a new user to the network, or verifying that the backups executed properly. Usually, these are relatively simple processes which are less about problem solving, and more about following the same set of steps over and over until the desired goal is attained. It is in these situations that the (properly) lazy admin identifies a way to automate as much as possible these processes, so that he or she can get back to more brain-intensive work (this has the net effect of improving overall efficiency and value - see how laziness pays off in the end ? :) ) There are, of course, as many scripting and programming languages as there are grains of sand on a beach, but despite the many competitors and alternatives out there, Perl remains the language of choice for many Linux admins around the world. This is in no small part due to Perl's ability to manipulate data in a rapid, logical, and easily deployable manner - the most obvious example of this being the vaunted « Perl One-Liner ». example !</h3> There comes a time in every admin's life when they must take a bunch of text files, and systematically swap some of the text within with new data - commonly known as searching and replacing. You could certainly do this by hand using an editor or by using a relatively straightforward C program if you were so inclined. But there is another way - a better, smarter, lazier way : the Perl search & replace one-liner ! Let's take a look at the code, then break down each component. $ perl -p -i -e 's/oldandbusted/newhotness/' *.txt</code></pre> That's it, you're done - take a lap and hit the showers. So, what exactly just happened there ? We employed a classic and very common usage method in command-line Perl which can easily be remembered as « pie » : « -p » : In a nutshell, this tells Perl to loop through each line of input, then perform the desired action (in this case, the search & replace) against each of those lines.</li> « -i » : This instructs Perl to actually edit the input files directly (or « in place »), instead of just displaying the changes on the screen.</li> « -e » : This describes exactly one line of code - in this case, the search and replace regular expression...</li> « 's/old/new/' » : This is the regular expression (or « regex ») which Perl will use to perform the search & replace. (What's a regex ? Wikipedia has the answers you seek</a> !)</li> « *.txt » : The target filename - in this case, a simple glob. (What's a glob ? Wikipedia has the answer</a> !)</li> </ul> The key to this whole operation was the fourth bullet point - the regex. Don't worry if your regex-fu is not yet strong - this is just an example, and it could have been anything - the point is that Perl can be used to rapidly execute regular expressions on data in simple, easy to execute ways, such as the search & replace one-liner above. This sort of thing comes in handy on a daily basis, and thus, the perl one-liner is a powerful tool in the System Administrator's toolbox. For more one-liners, use the Google : http://www.google.fr/search?q=perl+one-liners</a> initrd, modules, and tools 2009-06-10T00:00:00+00:00 Today i thought i'd discuss initrd, how modules live within it, and an give a generic example of how it can be modified on a Fedora-based system. Along the way, i'll provide supplementary information about gzip, tar, and cpio. Let's begin ! initrd</h3> The term « initrd » is short for « initial ramdisk ». It refers to a file of the same name used by Linux systems during the boot process in order to load certain modules (drivers, and the like) prior to the full system being brought online. An example usage of initrd would be to load a module for a RAID card, so that the actual filesystem - and all of the delicious data contained within - can be accessed properly. As with most everything, the related Wikipedia entry</a> has more general information on the subject. There are (perhaps unfortunately) a number of ways in which initrds can be constructed and implemented. Generally speaking, the actual file is a compressed archive, which may contain further archives, filesystems, configuration files, and so forth. Over the years, different distributions have handled the initrd in different ways, such that one distro's method of constructing an initrd may be totally incompatible with that of another. In fact, even within a single distribution, different releases may alter the internals of the initrd - Fedora being a recent popular example of this. Fedora</h3> In the earlier days of Fedora, the initrd was a compressed gzip archive that contained a « squashfs » file, which was itself a compressed filesystem. If you wanted to modify the contents of the initrd, all you had to do was un-gzip the initial file, then mount the squashed filesystem within. If some or all of these terms are fuzzy for you, don't despair - i'll explain everything as we go along. The important thing you need to know now is that in Fedora 8, the nature of the initrd changed significantly. In order to work with the current Fedora initrd, you need to be familiar with a few terms, their related technologies, and their usage... ramdisk</h3> A « ramdisk » is, as the name implies, a sort of filesystem which has been loaded into RAM. Although it does not physically exist, it appears for all intents and purposes to be a normal disk, which has the advantage of being seamless to the user (and to the system). Ramdisks are handy because, compared to physical disks, they are fast. Within the context of the initrd, however, speed isn't a concern - it's the versatility of the ramdisk that is key. As in the module example i mentioned initially, the various bits and peices of hardware in a given computer often require special drivers to make them function properly. These drivers can be built directly into the kernel, but often, for reasons of ease of use and cross-compatibility, it's easier to put them into the initrd, and then load them via the initial ramdisk that gets built when the machine boots. modules, in general</h3> The basic idea behind a « loadable kernel module » (or just module for short) is, again, to add support for hardware, special filesystems, and the like, which the kernel doesn't natively understand. Modules can be loaded at any time, but for the most part, they are initialised at boot time, so that the kernel can interact with things like RAID, audio, or video cards, non-Linux filesystems (NTFS, for example), or even more esoteric items like robotic controllers and (the dreaded) software modems. As a general rule, modules themselves are single files which are referenced via one or more configuration files. We'll take a closer look at these ideas below. gzip (and tar, too !)</h3> Gzip is, along with « tar », arguably the most ubiquitous archival format pairing in the Linux world. Generally speaking, a gzip archive is a single file which has been compressed using the gzip binary. Commonly, gzip is used to compress a « tar » archive, which normally does not use compression, but which can contain any number of files in a single archive. Together, they are used to create « .tar.gz » or « .tgz » files, which are more or less the standard way to create compressed archives in Linux ; together, they're like « zip » or « rar » (or « arj », or « boo ») in the Windows world. (Wikipedia</a>) Within the context of the Fedora initrd, however, tar is not used - from the pair, only gzip is important. cpio</h3> It is a rare and glorious thing when ancient technology is updated and used constantly from day one. Take the lightbulb, for example. Or fire. Or « cpio » . It's basically like tar (mentioned above), in that it can be used to take any number of given files, and create a single archival file containing said files. Like tar, compression isn't part of the deal, and gzip can be used to compress a cpio archive as normal. While cpio has a long and storied history (it was first specified way back in 1977), over the years, it has increasingly replaced by tar in many implementations. Some time ago, Red Hat decided to use cpio (instead of tar) for their ultra-popular packaging format « RPM ». This sparked a renewed interest in cpio in many communities, including, as you may have already guessed, the Fedora team - it's now employed regularly in the initrd (among other things). (Wikipedia</a>) Ok, lets go !</h2> The first thing we'll need to do is copy the initrd to somewhere handy, like « /tmp », so that we can play with a copy . Never play with the original - it should be preserved somewhere in case our modifications cause the file to become unusable. The initrd will be named « initrd.img » - but what is it, exactly ? Let's find out : $ cp <original_initrd.img> /tmp/ $ cd /tmp $ file initrd.img initrd.img: gzip compressed data, from Unix, last modified: Fri Nov 2 15:54:12 2007, max compression</code></pre> Ah, it's our old friend, gzip. Since we know that gzip is a file compression tool, we know that initrd.img contains a file - let's extract it : $ gzip -d initrd.img gzip: initrd.img: unknown suffix -- ignored</code></pre> Uh oh ! By default, gzip doesn't handle files with suffixes it doesn't understand (i.e. « .gz » ). The easiest way to deal with this is simply to add the appropriate suffix, then try again : $ mv initrd.img initrd.img.gz $ gzip -d initrd.img.gz</code></pre> This gives us a single file (as we expected) : initrd.img . Though it has the same name as the file we started with, it's not the same data at all. This initrd.img is larger than the original, and most importantly, it's a different type of file : $ file initrd.img initrd.img: ASCII cpio archive (SVR4 with no CRC)</code></pre> Ah ha ! It's a cpio archive - the first of two we'll encounter during this process. Note the trailing information, that it's an « SVR4 » archive, as this is important for the next step : unarchiving. Unpacking the initial cpio archive</h3> As i mentioned above, cpio has a long and storied history, which is reflected in the myriad of arguments and options that can be provided during normal usage. The command we'll be using at this stage will be : $ cpio -i -d -H newc -F <file> --no-absolute-filenames</code></pre> The arguments are, in order : -i : « copy-in » is more or less like saying « extract » in the parlance of cpio.</li> -d : « make directories » allows cpio to re-create the directory structure in the archive, instead of just unpacking everything to the same place.</li> -H newc : « format type » defines the particular format used by this cpio archive. Over the course of cpio's history, numerous formats have been used for various reasons : « newc » is an SVR4-based format that allows for large files and filesystems. Remember the archive type from above ? This is why we needed to make note of it.</li> -F : « file filename » simply indicates the file we want to work with.</li> --no-absolute-filenames : as the name suggests, this forces cpio to unpack the contents relative to where we executed the command. Basically speaking, it prevents the possibility of cpio trying to write to our root filesystem, which could have disastrous results.</li> </ul> For more information about cpio, simply consult the GNU cpio manual</a>. First we must make a directory in which to unpack the archive, then we can proceed : $ mkdir /tmp/initrd $ cd /tmp/initrd $ cpio -i -d -H newc -F ../initrd.img --no-absolute-filenames 16442 blocks</code></pre> The number of « blocks » refers to the amount of data that was unpacked - it is a measurement, like bytes. This value may be different for you, as not all initrd's are created equal. the goods</h3> A simple « ls » reveals the contents of the initrd. Astute readers will notice that it looks very familiar - this is not a coincidence - it looks like the root of a normal filesystem : $ ls -l total 36 lrwxrwxrwx 1 dan dan 4 2009-06-10 16:37 bin -> sbin drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 dev drwxrwxr-x 3 dan dan 4096 2009-06-10 16:37 etc lrwxrwxrwx 1 dan dan 10 2009-06-10 16:37 init -> /sbin/init drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 modules drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 proc drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 sbin drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 selinux drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 sys drwxrwxr-x 2 dan dan 4096 2009-06-10 16:37 tmp drwxrwxr-x 6 dan dan 4096 2009-06-10 16:37 var</code></pre> The directory we're interested in is « modules » : $ cd modules $ ls -l total 5128 -rw-r--r-- 1 dan dan 12904 2009-06-10 16:37 module-info -rw-r--r-- 1 dan dan 137235 2009-06-10 16:37 modules.alias -rw-r--r-- 1 dan dan 4971479 2009-06-10 16:37 modules.cgz -rw-r--r-- 1 dan dan 37966 2009-06-10 16:37 modules.dep -rw-r--r-- 1 dan dan 60204 2009-06-10 16:37 pci.ids</code></pre> Aah, delicious modules, at last ! But wait - what's that « .cgz » file ? $ file modules.cgz modules.cgz: gzip compressed data, from Unix, last modified: Fri Nov 2 15:54:07 2007, max compression</code></pre> Another gzip - but the suffix gives us another hint. Just as a .tgz is a gzip'd tar archive, a .cgz indicates a gzip'd cpio archive. Of course, as before, we'll need to rename this file in order to work with it : $ cp modules.cgz /tmp/modules.gz $ cd /tmp $ gzip -d modules.gz $ file modules modules: ASCII cpio archive (SVR4 with CRC)</code></pre> As before, we must create a working directory into which we'll unpack the cpio archive. Be sure to take note of the cpio arguments - there's been a change : $ mkdir /tmp/work $ cd /tmp/work $ cpio -i -d -H crc -F ../modules --no-absolute-filenames 25396 blocks</code></pre> How did i know to use a different format ? Look again at the file output for modules above, then compare it to the first cpio archive we dealt with. The difference is subtle, but critical - the cpio manual (linked above) lists them all, just in case. Ok, let's take a look at what we've got. $ tree -d . `-- 2.6.23.1-42.fc8 `-- i586</code></pre> Here, the string « 2.6.23.1-42.fc8 » refers to the kernel version that the initrd is built for, and the « i586 » refers to the architecture. Yours may be different (i randomly selected one that was hanging around on my test system), and that's OK, because the principles are still the same. For now, you'll note that the i586 directory is full of « .ko » files - these are the module files themselves. adding the new module</h3> The module that you want to add will be a .ko file as well. You may have been supplied with this file directly, or perhaps you compiled it from source (which is outside of the scope of this document for today). Regardless, as you may have guessed, you'll want to put it along with all the other modules : $ cp /tmp/new_module/network_card.ko /tmp/work/2.6.23.1-42.fc8/i586/</code></pre> With that in place, we'll re-pack cpio archive, so that it can be inserted back into the initrd. Through the magic of re-direction, we'll accomplish this all in one swift command : $ cd /tmp/work $ find 2.6.23.1-42.fc8 | cpio -o -H crc | gzip > /tmp/modules.cgz 25590 blocks</code></pre> Whoah ! Let's break that down : « find » was used to create a list of all the files with their relative directories. Run « find » by itself to get a better idea of what this means.</li> That list was then piped to cpio, where « -o » indicated that a new archive should be created (« copy-out » in the cpio parlance).</li> That was then directly passed to gzip, which ultimately outputted the gzip'd cpio file « modules.cgz » . Simple !</li> </ul> Now we can copy this shiny new modules.cgz into the initrd tree : $ cp /tmp/modules.cgz /tmp/initrd/modules/ cp: overwrite `/tmp/initrd/modules/modules.cgz'? y</code></pre> At this point, there's a very good chance that you'll need to modify a handful more items related to modules - in particular, the other files in the modules directory : $ cd /tmp/initrd/modules $ ls -l total 5228 -rw-r--r-- 1 dan dan 12904 2009-06-10 16:37 module-info -rw-r--r-- 1 dan dan 137235 2009-06-10 16:37 modules.alias -rw-r--r-- 1 dan dan 5074769 2009-06-10 17:35 modules.cgz -rw-r--r-- 1 dan dan 37966 2009-06-10 16:37 modules.dep -rw-r--r-- 1 dan dan 60204 2009-06-10 16:37 pci.ids</code></pre> There's our new modules.cgz, as you can see from the timestamp difference, along with the other items from before. Those other items are all text files which contain various bits and pieces of information related to the modules contained in modules.cgz . As this is a generic sort of post, i can't go into specifics about how each of these files would be modified for any actual module ; that said, a general overview of each of these items is most certainly in order : « module-info » : This file contains a an alphabetical list of each of the modules, with two lines per module that describe the type of device the module affects, and a verbose (i.e. human-readable) description of the module. It is the most simplistic of the four, and the one most easily read by a human. Example :</li> </ul> e1000 eth "Intel(R) PRO/1000 Network Driver"</code></pre> In this case, the module is named « e1000» , it affects Ethernet ( « eth » ) devices, and is an Intel gigabit NIC driver. « modules.alias » : Under normal usage, hardware will expose certain information to the kernel that helps the kernel to properly identify and assign said hardware. This is done via a sort of identification string, which contains (in machine-readable format) such things as the bus used by the hardware, the vendor and device id's, and other such sundry items. Normally, the system takes care of populating this file on its own, but in the case of the initrd, this doesn't happen. As a result, it is highly likely that you'll need to insert the identification string for your hardware manually ; hopefully, your module came with some documentation to help with this. If your module didn't come with the appropriate documentation, you could always try putting the hardware into an already-installed system, and then poking around for it manually. More information is available from the ArchLinux wiki</a>.</li> This file, since it's normally generated automatically, and used behind the scenes, is the most daunting of the four. Once you get a handle on how the string is built, however, it stops being scary, and starts being merely irritating. I'm not going to lie - this file is tough. Google is your friend on this one !</li> Example :</li> </ul> </li> </ul> alias pci:v00008086d0000108Bsv*sd*bc*sc*i* e1000</code></pre> « modules.dep » : This lists the « dependencies » for each module - effectively, it's a list of what modules the other modules depend on. Sometimes modules are designed to operate completely independently (such as the above-noted e1000 module), while other times they build on each other, with each module providing certain functionalities used by the next. This file builds those relationships. Example :</li> </ul> msdos: fat</code></pre> « pci.ids » : Finally, can you guess what's listed in this file ? If you guessed « PCI IDs », you're right ! This file is in the same genre as modules.alias, in that it contains a list of hardware identifiers that are used by the kernel in order to properly assign drivers and so forth. Again, as with modules.alias, this data is not necessarily obvious, and your hardware documentation should list the appropriate identifiers. In the event that your documentation doesn't have this information, there are two excellent sites that contain information on known PCI IDs : the PCI ID Repository</a>, and the PCI Vendor and Device List</a>. Not exactly bedtime reading, i know, but they're both life-savers !</li> Example :</li> </ul> </li> </ul> 101e American Megatrends Inc. 1960 MegaRAID 9010 MegaRAID 428 Ultra RAID Controller 9060 MegaRAID 434 Ultra GT RAID Controller</code></pre> Once these files have been properly updated, the only step that remains is re-packaging everything into a proper initrd.img ! new initrd.img</h3> Here in the final step we'll leverage the power of re-direction and create our initrd.img in one fell swoop : $ cd /tmp/initrd $ find . | cpio -o -H newc | gzip > /tmp/initrd.img $ file /tmp/initrd.img /tmp/initrd.img: gzip compressed data, from Unix, last modified: Wed Jun 10 19:39:51 2009</code></pre> And there you have it, a shiny new initrd.img, ready for use in your PXE bootstrap, disk image, or wherever else such things are needed. Aren't sure what a PXE bootstrap is ? Not sure why you'd want to use a disk image ? Stay tuned, fair reader, all will be revealed as we delve deeping into the mysterious land of the Linux System Administrator...}}}}}}}}}}}}}}}}}}

dan's tech blog

16 Years of DevOpsDays: A Look at the Numbers

Building a (mostly) reliable research assistant

Easily keep your local git repo up to date

Presentations and videos

Observability as code: dashboards

Devopsdays Berlin 2018; recap

So long, and thanks for all the fish.

Genesis: Terraforming a New Home for Firefox Crash Reporter

Handling extant resources in Terraform

examples</h1> In the examples below, the notation [...]</code> is used to indicate truncated output or data. Also note that the AWS cli tool is assumed to be configured and functional.</p>

Sysadvent 2014: What Sysadmins can learn from Community Management

Devopsdays Belgium 2014; recap

DevOpsDays Paris 2013, the event that was

hangops, european edition

HAProxy Puppet module (phrawzty remix)

LCA, the event that was

How to use Puppet like an Adult

Puppet at Mozilla, the podcast

quickly generate an encrypted password

Elasticsearch backup strategies

Send your logs to the cloud; Loggly vs. Papertrail

Heavyweight tilt : GitHub vs. Bitbucket

Improvements in Cassandra 1.0, briefly stated

Nagios plugin to parse JSON from an HTTP response

RabbitMQ plugin for Collectd

how to use the Distributed Numeric Assignment (DNA) plug-in in 389 Directory Server

CPAN RPMs in RHEL / CentOS : generation, conflict, and solutions

workaround for slow shared folders in Virtualbox 3.x

how to deal with broken time zones during a CentOS 5.3 kickstart

load balancing, or, « how i learned to love a piranha »

all together now !</h3> At this point we've now explored each key item that is necessary to make this whole front-end infrastructure work, but it is perhaps not quite clear how it all works together. So, let's take a step back for a moment and review :</p>

oh, p.s., one last thing...</h4> In case you're wondering how to keep your LVS configuration file synchronised across both of the load balancers, one way to do it would be with a network-aware filesystem - POHMELFS</a>, for example. ;-)</p>

it's coming !

where to specify ethtool options in Fedora

(complex) partitioning in kickstart

getting Dia to give you a pdf

pohmelfs pt. 2, return of pohmelfs !

setting the from address in GNU mail

pohmelfs update

force disk geometry with sfdisk

pohmelfs - the network filesystem of the future !

build & install the kernel</h3> From here, all that's left is to let it build - depending on your hardware, this can take a while. Patience, grasshopper.</p>

how to be properly lazy, with perl !

initrd, modules, and tools

examples</h1>
In the examples below, the notation `[...]</code> is used to indicate truncated output or data. Also note that the AWS cli tool is assumed to be configured and functional.</p>`

oh, p.s., one last thing...</h4>
In case you're wondering how to keep your LVS configuration file synchronised across both of the load balancers, one way to do it would be with a network-aware filesystem - POHMELFS</a>, for example. ;-)</p>