OONI Probe CLI v3.10.0-beta.3

iOS

pod "oonimkall", :podspec => "https://github.com/ooni/probe-cli/releases/download/v3.10.0-beta.3/oonimkall.podspec"

Android

implementation "org.ooni:oonimkall:2021.05.13-070256"

Generic build instructions

go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

These are the changes since v3.10.0-beta.2:

• ✨ we now publish linux/386 and linux/armv7 Debian packages.

• 🛠️ rewrite build script ./make (python3) as an executable Makefile named ./mk as a way to reduce the complexity of what we need once we have gained a better understanding of what we need to do and given up on automatically installing and configuring stuff on Windows for now (which is more difficult than expected, and honestly we are okay with cross-compiling).

• 🛠️ While there, simplify ./make / ./mk and sort out (1) what to do to ensure we pin the right version and we can easily update.

• 🛠️ since two weeks have passed, re-run the whole release workflow (#345) to update dependencies, etc.

OONI Probe CLI v3.10.0-beta.2

iOS

# not available

Android

// not available

Generic build instructions

go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

These are the changes since v3.10.0-beta.1:

• ✨ we now build and publish debian/arm64 packages

• 👀 the ./make tool is now used for all builds and all the previous build scripts are gone

• 🛠️ we now expose to the desktop app whether a measurement has been submitted

• 🐛 make sure we always run a tor (or tor.exe) binary that is in PATH (see https://blog.golang.org/path-security)

OONI Probe CLI v3.10.0-beta.1

iOS

pod "oonimkall", :podspec => "https://github.com/ooni/probe-cli/releases/download/v3.10.0-beta.1/oonimkall.podspec"

Android

implementation "org.ooni:oonimkall:2021.04.29-120054"

Generic build instructions

go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

These are the changes since v3.10.0-beta:

• 🛠️ new script (./make -t ios) to create iOS releases

• 🛠️ update documentation and remove all references to Bintray based builds

• 🛠️ remove legacy scripts for creating iOS/Android releases

OONI Probe CLI v3.10.0-beta

iOS

N/A

Android

implementation 'org.ooni:oonimkall:2021.04.28-095627'

Generic build instructions

go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

These are the changes since v3.10.0-alpha:

• 🛠️ new script (./make -t android) to create Android releases

• 🛠️ stop trying to upload android/iOS packages at bintray

• 🛠️ update dependencies, user agents, geoip databases, bundled certificates

OONI Probe CLI v3.10.0-alpha

iOS

N/A

Android

implementation 'org.ooni:oonimkall:2021.04.19-132002'

Generic build instructions

go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

These are the changes since v3.9.2:

• ✨ proxy: 👀 start redesigning tunnel and proxy code. We now support a build mode (-tags ooni_psiphon_config) where you can supply an encrypted Psiphon config file and its decryption key. These are currently private files. When building in this mode, you can always use the --tunnel=psiphon feature of the engine to create a Psiphon tunnel that speaks to the OONI backend. However, when not using -tags ooni_psiphon_config it is not possible anymore to use --tunnel=psiphon (which seems a good compromise because we are building OONI ourselves for the places with the most censorship). For now, only miniooni fully supports this new feature. We are working to bring these proxy enhancements to mobile devices as well as to the ooniprobe command.

• 🐛 geolocate: we used to avoid resolver identification when using a proxy. We clarified in ooni/probe#985 (comment) that this choice was not correct and relied on a wrong assumption (i.e., that the proxy was also used for measurements). So, since this release, we do perform resolver identification when using a proxy.

• 🛠️ pkg/oonimkall: add support for specifying the proxy URL (where psiphon:/// is psiphon with embedded configuration and socks5://127.0.0.1:9050 is the format required for a SOCKS5 proxy on 127.0.0.1:9050) and the tunnel directory (i.e., the directory where to store persistent data required by tunnels such as "tor" and "psiphon").

• 🛠️ ios: start migrating build script off bintray. We now publish iOS binaries along with probe-cli releases at GitHub. The process is currently manual. We will invest some time in subsequent releases to automate it again. The build-ios.bash script now includes the generation of a podspec file. The publish-ios.bash script is currently broken and needs to be replaced/removed,

• 🛠️ android: start migrating build script off bintray. We now publish Android binaries at Maven Central. The process is currently manual. We will invest some time in subsequent releases to automate it again. The build-android.bash script now includes PGP signing and generation of a bundle.jar for Maven Central. The publish-android.bash script is currently broken and needs to be replaced/removed.

• 🛠️ engine: we now create a directory under the OONI "home" directory (~/.ooniprobe on Unix) to store persistent information required by session tunnels like psiphon and tor. This allows us to much more quickly create such tunnels after they have been created once. That is, using tunnels for session operations is now much more efficient. This "tunnels" directory is created by mobile probes, by CLI probes, and by miniooni in their respective "home" directory. The "tunnels" directory is named tunnel for miniooni and ooniprobe and will be named tunnel also for the mobile probes. Every kind of tunnel will create a specifically named child directory, so to obtain, e.g., $OONI_HOME/tunnel/psiphon.

• 🛠️ new dependencies: filippo.io/age to decipher Psiphon's embedded configuration file and github.com/armon/go-socks5 to implement a fake session tunnel that is very useful to increase our testing coverage.

• 🛠️ miniooni: the --tunnel=foo option is now syntactic sugar for --proxy=foo:///. We have changed the command line parsing code such that it's now an error to specify --tunnel and --proxy together.

• 🛠️ engine: the NewSession factory will now start a tunnel if instructed to do so. This allows us to simplify the codebase, which was more complex when tunnels needed to be started manually at a later time. However, this means that NewSession now requires a context.Context argument, so we needed to change a bunch of places in the internal sources.

• 🛠️ engine: merge internal/engine/internal/torx and internal/engine/internal/psiphonx and internal/engine/internal/tunnel into the internal/engine/tunnel package.

• 🛠️ engine: removed the OrchestraClient abstraction and move the related implementation bits into Session, which allows us to move more quickly because it simplifies mocking the requirements of what depends on a Session-like interface.

OONI Probe CLI v3.9.2

🐛 This is like 3.9.1 plus 818bf2c (which fixes ooni/probe#1435) and c5f52d5 (which fixes ooni/probe#1436).

OONI Probe CLI v3.9.1

🐛 This is a patch release consisting of v3.9.1 plus a fix to disable maxRuntime when the experiment is not "web_connectivity" (6306c09, forward ported to master as #302). With this fix in, we don't break probe-desktop's progress bars.

OONI Probe CLI v3.9.0

iOS

pod 'oonimkall', :podspec => 'https://dl.bintray.com/ooni/ios/oonimkall-2021.04.01-183645.podspec'  # using bintray (will break)
pod 'oonimkall', :podspec => 'https://github.com/ooni/probe-cli/releases/download/v3.9.0/oonimkall.podspec'  # using github (should work)

Android

implementation 'org.ooni:oonimkall:2021.04.01-183918'    // broken (still using bintray)
implementation 'org.ooni:oonimkall:2021.04.07-180801'    // works (maven central, use this one)

Generic build instructions

go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

• ✨ embed resources via a Go dependency: go run ./internal/cmd/getresources no longer required. The getresources command itself is now a no-op that tells you about this change. We will remove it at the end of 2021.

• ✨ ooniprobe run unattended is no longer limited to 10 URLs per run: we now use a more advanced OONI backend API called check-in that allows us to better provision work to probes running in background, taking into account their state and what has been tested so far by other probes in the same country and ASN

• ✨ the riseupvpn experiment has been improved to reduce false positives (thanks @cyBerta!)

• ✨ the webconnectivity experiment now includes network_events and tls_handshake events generated during the TCP/TLS phase of the experiment. (You can read more on this in the updated spec: ooni/spec#214)

• 👀 ❗ the websites_url_limit option has been replaced by websites_max_runtime to align the cli with the desktop and mobile probes. If websites_url_limit is nonzero in the config, we will nudge you to migrate over to websites_max_runtime and we will convert it to website_max_runtime where every URL in websites_url_limit counts a five seconds. The warning message and the automatic conversion will not be removed until the end of 2021. (Note the ooniprobe run unattended will not honour websites_max_runtime because in that case the run time is dynamically controlled by the server, which serves us a small enough number of URLs.)

• 👀 ❗likewise, miniooni now rejects --limit: you should use --max-runtime instead. (Because miniooni is an internal tool, we feel comfortable with breaking its command line usage without automatically converting --limit as --max-runtime, as we do for ooniprobe). After 2021-11-01, miniooni will stop telling you explicitly that --limit has been removed in favour of --max-runtime.

• 👀 ❗ooniprobe --batch now emits output on the standard output, to avoid mixing JSON output with non-JSON warnings that may be emitted by third party libraries (e.g., lucas-clemente/quic-go)

• 👀 ❗because now we embed resources directly into the binary using a Go dependency, the go.mod of a release can be used to reconstruct which database is embedded. So we don't include anymore an annotation with the assets version number.

• 👀 ❗ the signal experiment has been moved to the im nettest group

• 🐛 sessionresolver: make sure we don't skip the system resolver and http3 resolvers when we have no configured proxy (this bug was introduced in #250 and fixed by #264)

• 🛠️ we are not storing resources (e.g. MaxMind databases) to disk anymore: they are now embedded inside the binary as uncompressed blobs of data. We will automatically cleanup the directory assets inside the $OONI_HOME where we used to store the resources. We will just delete known file names within the directory (asn.mmdb, country.mmdb, and ca-bundle.pem) and then attempt to rmdir the directory. In the unlikely case in which you put some of your files in there, we will fail to remove the assets directory and your files will still be there. See the documentation and the implementation for more info on this new resources design.

• 🛠️ we updated dependencies, MaxMind databases, and other bundled resources

• 🛠️ improved (currently unused) auto-generated ooapi code to speak with the OONI backend. We will switch over to this code when we feel like it's production ready and feature complete to replace existing code.

• 🛠️ add new experimental API in pkg/oonimkall that we would like to test for running nettests directly from Go rather than passing through the MK-compatible JSON API

• 🦺 we added some extra code to ensure we are not going to test URLs outside of the user selected categories. The code has always behaved as intended so far, but it's nice to increase the codebase robustness by adding extra safety nets.

OONI Probe CLI v3.8.0

iOS

pod 'oonimkall', :podspec => 'https://dl.bintray.com/ooni/ios/oonimkall-2021.03.15-152119.podspec'

Android

implementation 'org.ooni:oonimkall:2021.03.15-152735'

Generic build instructions

go run ./internal/cmd/getresources
go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

• 🐛 👀 rename stun_reachability to stunreachability

• 🐛 👀 stunreachability: fix goroutine spin and memleaks (see ooni/probe#1403)

• 🐛 sessionresolver: honor proxy settings

• 🐛 geolocate: disable proxy when discovering IP address

• 🔨 android: stop pinning to specific NDK version

• 🔨 miniooni: introduce --version and --limit

• 🔨 refactor, document, and simplify oonimkall code

OONI Probe CLI v3.7.0

iOS

pod 'oonimkall', :podspec => 'https://dl.bintray.com/ooni/ios/oonimkall-2021.03.08-125957.podspec'

Android

implementation 'org.ooni:oonimkall:2021.03.08-130053'

Generic build instructions

go run ./internal/cmd/getresources
go build -v $extraBuildFlags ./cmd/ooniprobe

Changes

• 🔨 👀 We moved the Signal messenger experiment into the experimental nettest group. So, if you wanna run this experiment, you now need to run ooniprobe run experimental. We will keep this experiment into this group until the desktop app has a UI for it. We don't want the desktop app to run experiments for which there is no UI.

• ✨ We added dnscheck and stunreachability to the experimental nettest group. This is part of an effort to start exposing more experimental tests to users. For now, this group is only available to CLI users.

• 🐛 webconnectivity now fails less badly when asked to test for https://1.1.1.1. Before the whole test failed, now only the control step fails because the legacy control endpoint does not support this use case. This enhancement would still enable us to draw conclusions regarding whether https://1.1.1.1 is blocked, even though the UI will show a failed experiment.

• 🐛 windows: ooniprobe run unattended now only tests a handful of URLs, like we do on macOS.

• 🔨 we have merged code that improves how we communicate with out backend. This code is currently not used by ooniprobe but its presence already allows us to check whether our definition of the API spec is in sync with the backend. We will take advantage of this improvement in subsequent releases, when we'll start using the new code.