44 stories

Ever The Skeptic

1 Comment

Ads by Project Wonderful! Your ad could be here, right now.


Read the whole story
35 days ago
"One of the very few downsides of being as powerful as we are is [that] there are no credentials of benevolence we could not falsify."
Share this story

How to Get Vindication


For years, YEARS, I told that story, and I got called a liar more times than I can count. I always wondered, why would I want to make a story like that up? Did they think having seen this ridiculous, disgusting spectacle somehow made me feel like a big man?

“You think you’re cool, with your sports car and your successful career? Well wait until you hear my tale of gore, degradation, and animal husbandry! Then we’ll know who’s cool!”

Anyway, I’ll include a link to the relevant clip of Dirty Jobs, but I don’t recommend that you watch it.


Read the whole story
67 days ago
TIL a sharp knife is "a lot of special equipment".
Share this story
2 public comments
70 days ago
Mike Rowe gave a TED Talk where he talked about being oh so wrong regarding the sheep castration process. Seems like the rubber band method is an inhumanely slow process.
"And I said, 'Why would you do it this way?' And before I even let him explain, I said, 'I want to do it the right way, with the rubber bands.'"
78 days ago
My answer would be "oh yes I know about that". Ah, farming... *shudders*
FourSquare, qv
78 days ago
Luckily, my relatives spent the money on the special equipment. Which in this case is just a tool for expanding a tiny rubber band.

The Yahoo-email-search story is garbage

1 Comment
Joseph Menn (Reuters) is reporting that Yahoo! searched emails for the NSA. NSA. The details of the story are so mangled that it's impossible to say what's actually going on.

The first paragraph says this:
Yahoo Inc last year secretly built a custom software program to search all of its customers' incoming emails
The second paragraph says this:
The company complied with a classified U.S. government demand, scanning hundreds of millions of Yahoo Mail accounts
Well? Which is it? Did they "search incoming emails" or did they "scan mail accounts"? Whether we are dealing with emails in transmit, or stored on the servers, is a BFD (Big Fucking Detail) that you can't gloss over and confuse in a story like this. Whether searches are done indiscriminately across all emails, or only for specific accounts, is another BFD.

The third paragraph seems to resolve this, but it doesn't:
Some surveillance experts said this represents the first case to surface of a U.S. Internet company agreeing to an intelligence agency's request by searching all arriving messages, as opposed to examining stored messages or scanning a small number of accounts in real time.
Who are these "some surveillance experts"? Why is the story keeping their identities secret? Are they some whistleblowers afraid for their jobs? If so, then that should be mentioned. In reality, they are unlikely to be real surveillance experts, but just some random person that knows slightly more about the subject than Joseph Menn, and their identities are being kept secret in order to prevent us from challenging these experts -- which is a violation of journalistic ethics.

And, are they analyzing the raw information the author sent them? Or are they opining on the garbled version of events that we see in the first two paragraphs.

The confusion continues:

It is not known what information intelligence officials were looking for, only that they wanted Yahoo to search for a set of characters. That could mean a phrase in an email or an attachment, said the sources, who did not want to be identified.
What the fuck is a "set of characters"??? Is this an exact quote for somewhere? Or something the author of the story made up? The clarification of what this "could mean" doesn't clear this up, because if that's what it "actually means", then why not say this to begin with?

It's not just technical terms, but also legal ones:

The request to search Yahoo Mail accounts came in the form of a classified edict sent to the company's legal team, according to the three people familiar with the matter.
What the fuck is a "classified edict"? An NSL? A FISA court order? What? This is also a BFD.

We outsiders already

What outsiders
know about the NSA/FBI's ability to ask for strong selectors(email addresses). What what we don't know about is their ability to search allemails, regardless of account, for arbitrary keywords/phases. If that's what's going on, then this would be a huge story. But the story doesn't make it clear that this is actually what's going on -- just strongly implies it.

There are many other ways to interpret this story. For example, the government may simply be demanding that when Yahoo satisfies demands for emails (based on email addresses), that it does so from the raw incoming stream, before it hits spam/malware filters. Or, they may be demanding that Yahoo satisfies their demands with more secrecy, so that the entire company doesn't learn of the email addresses that a FISA order demands. Or, the government may be demanding that the normal collection happen in real time, in the seconds that emails arrive, instead of minutes later.

Or maybe this isn't an NSA/FISA story at all. Maybe the DHS has a cybersecurity information sharing program that distributes IoCs (indicators of compromise) to companies under NDA. Because it's a separate program under NDA, Yahoo would need to setup a email malware scanning system separate from their existing malware system in order to use those IoCs. (@declanm's stream has further variations on this scenario).

My point is this: the story is full of mangled details that really tell us nothing. I can come up with multiple, unrelated scenarios that are consistent with the content in the story. The story certainly doesn't say that Yahoo did anything wrong, or that the government is doing anything wrong (at least, wronger than we already know).

I'm convinced the government is up to no good, strong arming companies like Yahoo into compliance. The thing that's stopping us from discovering malfeasance is poor reporting like this.

this is that reporters are so shitty at reporting the details.

Read the whole story
140 days ago
RG may not be aware, but documents must be scanned before terms in them can be searched.
Share this story

9/11 Fetishism Dishonors Those Who Died That Day

1 Comment
Q: What's the difference between 9/11 and a cow?
A: The government can't milk a cow for 15 years and counting.

I have a confession to make: I'm udderly (see what I did there?) sick of 9/11 fetishism.

On December 7, 1956, was the U.S. government still using the Pearl Harbor attacks to justify why key parts of the Constitution should be considered invalid? Of course not – even though that was the height of the Cold War, when our country faced a no-joke existential threat: the Soviet Union had enough nuclear weapons to make humanity go extinct.

About a year ago, CNN ran a documentary series called “The Seventies,” with one episode dedicated to the rise of terrorism in that decade. And that episode genuinely shocked me. I of course have only childish memories of that time – I vaguely recall wondering why grown-ups who flew on airplanes thought saying “Hi” to someone named “Jack” was such a terrible, awful, no-good thing to do – but now that I'm an adult looking back on that era, I can see why Americans (and Westerners in general) would've been justified in thinking the entire civilized world was falling apart. There were some particularly bad weeks wherein there would be major airline hijackings every day – not only in the United States, but throughout the Western (non-Communist) world. Bombings were downright commonplace too, as were kidnappings and assassinations of even the rich and powerful. Yet America was able to get through all of that without gutting the Bill of Rights and making a holy fetish out of fear.

What the hell happened, to make early 21st-century America behave so much more helplessly, cowardly and reactionarily, compared to the craphole America of the 1970s? In the 70s, we'd just lost our first war (Vietnam); our economy was in the toilet (stagflation); there was an actual no-joke nationwide gasoline shortage thanks to the Arab oil embargo; the Cold War meant we had to live with the genuine existential fear of knowing "Our number-one enemy on the world stage has a nuclear arsenal sufficient to wipe us out in 15 minutes" -- by any objective measure, 1970s America was in far worse shape than the America of Sept. 10, 2001. So why did the latter America fail so spectacularly, when first it faced adversity?

It's not as though the majority of the Powers That Be, circa 2001, were entirely different people or an entirely different generation than those from the 1970s -- many of the young or middle-aged PTBs from the 70s still held power in 2001. And every power-broker in 2001 was old enough to have personal memories of the 1970s, even if they were not yet members of the "leadership class" back then. "Learning from history" for them would not have entailed reading books about how things were before their time; it would simply be a matter of remembering their own personal experience.

I'm sick and tired of being told it's my I have a patriotic duty to be afraid, and that defending the constitution is synonymous with dishonoring those who died fifteen years ago. afraid. Nine-eleven! Never forget! – now bend over and let the TSA fondle your genitalia before allowing you on an airline flight. Nine-eleven! Never forget! – now don't complain about the NSA spying on your electronic communications, but do tell yourself Edward Snowden is a traitor for bringing such unconstitutional actions this to light. Nine-eleven! Never forget! Never stop being afraid! Because if you're not still terrified fifteen years later, that supposedly somehow means the terrorists have won.
Read the whole story
147 days ago
Yes this is all baffling. Have some self-respect, Americans.
Share this story

Moon Shapes

6 Comments and 14 Shares
Whenever I see a picture of the moon where the points go more than halfway around, I assume it's being eclipsed by one of those Independence Day ships and interpret the rest of the image in light of that.
Read the whole story
154 days ago
Randall should visit the Southern Hemisphere to view the crescent moon.
152 days ago
Share this story
4 public comments
154 days ago
i was kind of hoping for a seveneves reference.
iPhone: 44.910601,-93.336335
154 days ago
What, no "That's no moon" joke?
154 days ago
Whenever I see a picture of the moon where the points go more than halfway around, I assume it's being eclipsed by one of those Independence Day ships and interpret the rest of the image in light of that.
154 days ago
Spotter's guide.

Python Packaging Is Good Now

1 Comment

Okay folks. Time’s up. It’s too late to say that Python’s packaging ecosystem terrible any more. I’m calling it.

Python packaging is not bad any more. If you’re a developer, and you’re trying to create or consume Python libraries, it can be a tractable, even pleasant experience.

I need to say this, because for a long time, Python’s packaging toolchain was … problematic. It isn’t any more, but a lot of people still seem to think that it is, so it’s time to set the record straight.

If you’re not familiar with the history it went something like this:

The Dawn

Python first shipped in an era when adding a dependency meant a veritable Odyssey into cyberspace. First, you’d wait until nobody in your whole family was using the phone line. Then you’d dial your ISP. Once you’d finished fighting your SLIP or PPP client, you’d ask a netnews group if anyone knew of a good gopher site to find a library that could solve your problem. Once you were done with that task, you’d sign off the Internet for the night, and wait about 48 hours too see if anyone responded. If you were lucky enough to get a reply, you’d set up a download at the end of your night’s web-surfing.

pip search it wasn’t.

For the time, Python’s approach to dependency-handling was incredibly forward-looking. The import statement, and the pluggable module import system, made it easy to get dependencies from wherever made sense.

In Python 2.01, Distutils was introduced. This let Python developers describe their collections of modules abstractly, and added tool support to producing redistributable collections of modules and packages. Again, this was tremendously forward-looking, if somewhat primitive; there was very little to compare it to at the time.

Fast forwarding to 2004; setuptools was created to address some of the increasingly-common tasks that open source software maintainers were facing with distributing their modules over the internet. In 2005, it added easy_install, in order to provide a tool to automate resolving dependencies and downloading them into the right locations.

The Dark Age

Unfortunately, in addition to providing basic utilities for expressing dependencies, setuptools also dragged in a tremendous amount of complexity. Its author felt that import should do something slightly different than what it does, so installing setuptools changed it. The main difference between normal import and setuptools import was that it facilitated having multiple different versions of the same library in the same program at the same time. It turns out that that’s a dumb idea, but in fairness, it wasn’t entirely clear at the time, and it is certainly useful (and necessary!) to be able to have multiple versions of a library installed onto a computer at the same time.

In addition to these idiosyncratic departures from standard Python semantics, setuptools suffered from being unmaintained. It became a critical part of the Python ecosystem at the same time as the author was moving on to other projects entirely outside of programming. No-one could agree on who the new maintainers should be for a long period of time. The project was forked, and many operating systems’ packaging toolchains calcified around a buggy, ancient version.

From 2008 to 2012 or so, Python packaging was a total mess. It was painful to use. It was not clear which libraries or tools to use, which ones were worth investing in or learning. Doing things the simple way was too tedious, and doing things the automated way involved lots of poorly-documented workarounds and inscrutable failure modes.

This is to say nothing of the fact that there were critical security flaws in various parts of this toolchain. There was no practical way to package and upload Python packages in such a way that users didn’t need a full compiler toolchain for their platform.

To make matters worse for the popular perception of Python’s packaging prowess2, at this same time, newer languages and environments were getting a lot of buzz, ones that had packaging built in at the very beginning and had a much better binary distribution story. These environments learned lessons from the screw-ups of Python and Perl, and really got a lot of things right from the start.

Finally, the Python Package Index, the site which hosts all the open source packages uploaded by the Python community, was basically a proof-of-concept that went live way too early, had almost no operational resources, and was offline all the dang time.

Things were looking pretty bad for Python.


Here is we get to the point of this post - this is where popular opinion about Python packaging is stuck. Outdated information from this period abounds. Blog posts complaining about problems score high in web searches. Those who used Python during this time, but have now moved on to some other language, frequently scoff and dismiss Python as impossible to package, its packaging ecosystem as broken, PyPI as down all the time, and so on. Worst of all, bad advice for workarounds which are no longer necessary are still easy to find, which causes users to pre-emptively break their environments where they really don’t need to.

From The Ashes

In the midst of all this brokenness, there were some who were heroically, quietly, slowly fixing the mess, one gnarly bug-report at a time. pip was started, and its various maintainers fixed much of easy_install’s overcomplexity and many of its flaws. Donald Stufft stepped in both on Pip and PyPI and improved the availability of the systems it depended upon, as well as some pretty serious vulnerabilities in the tool itself. Daniel Holth wrote a PEP for the wheel format, which allows for binary redistribution of libraries. In other words, it lets authors of packages which need a C compiler to build give their users a way to not have one.

In 2013, setuptools and distribute un-forked, providing a path forward for operating system vendors to start updating their installations and allowing users to use something modern.

Python Core started distributing the ensurepip module along with both Python 2.7 and 3.3, allowing any user with a recent Python installed to quickly bootstrap into a sensible Python development environment with a one-liner.

A New Renaissance

I’m won’t give you a full run-down of the state of the packaging art. There’s already a website for that. I will, however, give you a précis of how much easier it is to get started nowadays. Today, if you want to get a sensible, up-to-date python development environment, without administrative privileges, all you have to do is:

$ python -m ensurepip --user
$ python -m pip install --user --upgrade pip
$ python -m pip install --user --upgrade virtualenv

Then, for each project you want to do, make a new virtualenv:

$ python -m virtualenv lets-go
$ . ./lets-go/bin/activate
(lets-go) $ _

From here on out, now the world is your oyster; you can pip install to your heart’s content, and you probably won’t even need to compile any C for most packages. These instructions don’t depend on Python version, either: as long as it’s up-to-date, the same steps work on Python 2, Python 3, PyPy and even Jython. In fact, often the ensurepip step isn’t even necessary since pip comes preinstalled. Running it if it’s unnecessary is harmless, even!

Other, more advanced packaging operations are much simpler than they used to be, too.

  • Need a C compiler? OS vendors have been working with the open source community to make this easier across the board:
    $ apt install build-essential python-dev # ubuntu
    $ xcode-select --install # macOS
    $ dnf install @development-tools python-devel # fedora
    C:\> REM windows
    C:\> start https://www.microsoft.com/en-us/download/details.aspx?id=44266

Okay that last one’s not as obvious as it ought to be but they did at least make it freely available!

  • Want to upload some stuff to PyPI? This should do it for almost any project:

    $ pip install twine
    $ python setup.py sdist bdist_wheel
    $ twine upload dist/*
  • Want to build wheels for the wild and wooly world of Linux? There’s an app4 for that.

Importantly, PyPI will almost certainly be online. Not only that, but a new, revamped site will be “launching” any day now3.

Again, this isn’t a comprehensive resource; I just want to give you an idea of what’s possible. But, as a deeply experienced Python expert I used to swear at these tools six times a day for years; the most serious Python packaging issue I’ve had this year to date was fixed by cleaning up my git repo to delete a cache file.

Work Still To Do

While the current situation is good, it’s still not great.

Here are just a few of my desiderata:

  • We still need better and more universally agreed-upon tooling for end-user deployments.
  • Pip should have a GUI frontend so that users can write Python stuff without learning as much command-line arcana.
  • There should be tools that help you write and update a setup.py. Or a setup.python.json or something, so you don’t actually need to write code just to ship some metadata.
  • The error messages that you get when you try to build something that needs a C compiler and it doesn’t work should be clearer and more actionable for users who don’t already know what they mean.
  • PyPI should automatically build wheels for all platforms by default when you upload sdists; this is a huge project, of course, but it would be super awesome default behavior.

I could go on. There are lots of ways that Python packaging could be better.

The Bottom Line

The real takeaway here though, is that although it’s still not perfect, other languages are no longer doing appreciably better. Go is still working through a number of different options regarding dependency management and vendoring, and, like Python extensions that require C dependencies, CGo is sometimes necessary and always a problem. Node has had its own well-publicized problems with their dependency management culture and package manager. Hackage is cool and all but everything takes a literal geological epoch to compile.

As always, I’m sure none of this applies to Rust and Cargo is basically perfect, but that doesn’t matter, because nobody reading this is actually using Rust.

My point is not that packaging in any of these languages is particularly bad. They’re all actually doing pretty well, especially compared to the state of the general programming ecosystem a few years ago; many of them are making regular progress towards user-facing improvements.

My point is that any commentary suggesting they’re meaningfully better than Python at this point is probably just out of date. Working with Python packaging is more or less fine right now. It could be better, but lots of people are working on improving it, and the structural problems that prevented those improvements from being adopted by the community in a timely manner have almost all been addressed.

Go! Make some virtualenvs! Hack some setup.pys! If it’s been a while and your last experience was really miserable, I promise, it’s better now.

Am I wrong? Did I screw up a detail of your favorite language? Did I forget to mention the one language environment that has a completely perfect, flawless packaging story? Do you feel the need to just yell at a stranger on the Internet about picayune details? Feel free to get in touch!

  1. released in October, 2000 

  2. say that five times fast. 

  3. although I’m not sure what it means to “launch” when the site is online, and running against the production data-store, and you can use it for pretty much everything... 

  4. “app” meaning of course “docker container” 

Read the whole story
196 days ago
Nice, crapping on NPM without ever having used it. Separation of Concerns is a *good* thing, kids.
Share this story
Next Page of Stories