Suggestion: use server-side swift to run integration tests

[colinmarc]

Hi, thanks for the great library!

Currently, running the integration test suite (which is fantastic) requires xcodebuild. It seems to me that it should be easy to use server-side swift for this, which would allow running the integration tests on different platforms (for example, linux CI runners).

[chinedufn]

No problem, glad you're poking around.

---

Questions:

• How difficult would it be to port the test suite from Xcode -> using swiftc directly (or whatever "server-side" Swift is)?

  • Why is it "easy"? What work would be required?

• Would all integration tests be possible to run outside of Xcode, or would we still need Xcode for some tests?

• Are there any trade-offs to ditching Xcode for our integration test suite?

---

Yeah, enabling folks that don't have Apple hardware to run the test suite would be great.

[colinmarc]

• How difficult would it be to port the test suite from Xcode -> using swiftc directly (or whatever "server-side" Swift is)?

I believe it would require no changes to the swift code, only to the test harness. I've never used XCTest without XCode, but googling around it seems to be possible.

• Why is it "easy"? What work would be required?

1. The rust-side code would have to be wrapped as a swift package instead of added directly to the harness xcproject. I believe there's already an example of this elsewhere in the repo.
2. The test runner itself would be a swift package rather than an xcode project. It would add the rust code as a dependency.

* Would all integration tests be possible to run outside of Xcode, or would we still need Xcode for some tests?

I don't know the answer to this, but given that the tests seem to be mostly pure swift, I can't imagine many would depend on xcode. Are there any tests using platform stuff like SwiftUI?

* Are there any trade-offs to ditching Xcode for our integration test suite?

Some people like xcode, and they would (probably) have to drop to the command line to run the integration test suite.

[colinmarc]

I believe there's already an example of this elsewhere in the repo.

I was incorrect about this, so this may be the tricky bit. It should be possible to wrap a static librust_library_foo.a file in a swift package, but I haven't done that before (without creating an xcframework first). The official documentation mentions system libraries only: https://github.com/swiftlang/swift-package-manager/blob/main/Documentation/Usage.md#requiring-system-libraries

[chinedufn]

Got it, thanks.

I just did some quick research.

Some Relevant Links

• Swift on Server
  • https://www.swift.org/documentation/server/
• Swift Testing
  • https://www.swift.org/documentation/server/guides/testing.html
  • https://github.com/swiftlang/swift-testing

---

Looks like anyone with the Swift CLI can run swift test to use Swift Testing, Swift's new test harness / tools.

---

Path forward:

• we create a new empty directory for our integration tests. We'll gradually migrate old tests to this new directory.

• migrate one test file, such as PrimitiveTests.swift

  swift-bridge/SwiftRustIntegrationTestRunner/SwiftRustIntegrationTestRunnerTests/PrimitiveTests.swift

  Lines 1 to 33 in 225380e

  	//
  	// PrimitiveTests.swift
  	// SwiftRustIntegrationTestRunnerTests
  	//
  	// Created by Frankie Nwafili on 10/3/22.
  	//
  	import XCTest
  	@testable import SwiftRustIntegrationTestRunner
  	/// Tests for generic types such as `type SomeType<u32>`
  	class PrimitiveTests: XCTestCase {
  	/// Run tests where Rust calls Swift functions that take primitive args.
  	func testRustCallsSwiftPrimitives() throws {
  	test_rust_calls_swift_primitives()
  	}
  	/// Run tests where Swift calls Rust functions that take primitive args.
  	func testSwiftCallsRustPrimitives() throws {
  	XCTAssertEqual(rust_double_u8(10), 20);
  	XCTAssertEqual(rust_double_i8(10), 20);
  	XCTAssertEqual(rust_double_u16(10), 20);
  	XCTAssertEqual(rust_double_i16(10), 20);
  	XCTAssertEqual(rust_double_u32(10), 20);
  	XCTAssertEqual(rust_double_i32(10), 20);
  	XCTAssertEqual(rust_double_u64(10), 20);
  	XCTAssertEqual(rust_double_i64(10), 20);
  	XCTAssertEqual(rust_double_f32(10.0), 20.0);
  	XCTAssertEqual(rust_double_f64(10.0), 20.0);
  	XCTAssertEqual(rust_negate_bool(true), false);
  	XCTAssertEqual(rust_negate_bool(false), true);
  	}
  	}

• Make the GitHub actions CI run the tests on Linux, macOS and Windows

• going forward new tests go in the new dir, and old tests will gradually get ported over

[chinedufn]

It should be possible to wrap a static librust_library_foo.a file in a swift package, but I haven't done that before (without creating an xcframework first)

A couple of our examples use swiftc to link a Rust native library into a Swift program. Would that do the trick here, or no?

swift-bridge/examples/async-functions/build.sh

Lines 1 to 14 in 225380e

	#!/bin/bash
	set -e
	THISDIR=$(dirname $0)
	cd $THISDIR
	cargo build -p async-functions
	swiftc -L ../../target/debug \
	-lasync_functions \
	-import-objc-header bridging-header.h \
	-framework CoreFoundation -framework SystemConfiguration \
	main.swift ./generated/SwiftBridgeCore.swift ./generated/async-functions/async-functions.swift

[colinmarc]

Make the GitHub actions CI run the tests on Linux, macOS and Windows

For what it's worth, this seems a bit wasteful/inconvenient to me. The value of the tests is that the generated swift code is correct, not that the test harness itself is portable.

A couple of our examples use swiftc to link a Rust native library into a Swift program. Would that do the trick here, or no?

Indeed, the important part here is to let swift know where the rust library lives and that it has to link to it.

I believe what would work is to follow these instructions: https://github.com/swiftlang/swift-package-manager/blob/main/Documentation/Usage.md#requiring-a-system-library-without-pkg-config

Here is an example from the internet of this being done with a static library: https://stackoverflow.com/questions/64958357/how-to-build-a-local-c-library-as-a-static-library-and-link-with-swift

The crucial part being this:

You must also run swift build with ... -Xlinker -L/dir/which/lib/resides

[chinedufn]

My understanding is that Swift isn't fully cross platform yet, meaning it's possible for swift-bridge to generate code that works on macOS but not on Linux, simply because something that seems cross platform isn't actually supported on Linux yet.

I vaguely recall trying to import Foundation a couple years ago and discovering that that didn't work on Linux. I think ... can't remember.

If the above is true, running the test suite on Linux and Windows would reduce the likelihood of us unknowingly making our generated code use features that aren't supported on all platforms.

I'd have to do more research on this, but if it isn't actually true then yeah running the test suite on one OS would be fine.

---

Ok great, looks like we have a the pieces we need.

[colinmarc]

I'm having trouble because the generated swift bridge code (from crates/swift-integration-tests) depends on swift code in the test harness, creating a circular dependency.

For now, I will create a separate crate which is just the rust code used by the "swift calling into rust" tests. Or is the circular dependency / two-way bridge scenario supposed to work?

[colinmarc]

I'm also getting some errors that are probably related to my swift version.

% swift -version
Swift version 6.0.3 (swift-6.0.3-RELEASE)
Target: x86_64-unknown-linux-gnu

/home/colinmarc/dev/swift-bridge/integration-tests/Sources/RustLib/Generated/SwiftBridgeCore.swift:871:14: error: non-final class 'RustString' cannot conform to 'Sendable'; use '@unchecked Sendable'
 869 | protocol SwiftBridgeGenericCopyTypeFfiRepr {}
 870 |
 871 | public class RustString: RustStringRefMut {
     |              `- error: non-final class 'RustString' cannot conform to 'Sendable'; use '@unchecked Sendable'
 872 |     var isOwned: Bool = true
 873 |

/home/colinmarc/dev/swift-bridge/integration-tests/Sources/RustLib/Generated/SwiftBridgeCore.swift:872:9: error: stored property 'isOwned' of 'Sendable'-conforming class 'RustString' is mutable
 870 |
 871 | public class RustString: RustStringRefMut {
 872 |     var isOwned: Bool = true
     |         `- error: stored property 'isOwned' of 'Sendable'-conforming class 'RustString' is mutable
 873 |
 874 |     public override init(ptr: UnsafeMutableRawPointer) {

[colinmarc]

One more thing - I also noticed that a few tests are using SwiftUI:

swift-bridge/SwiftRustIntegrationTestRunner/SwiftRustIntegrationTestRunner/ContentView.swift

Line 8 in 225380e

import SwiftUI

Those won't work on Linux, but we might be able to exclude them there.

[colinmarc]

I'm having trouble because the generated swift bridge code (from crates/swift-integration-tests) depends on swift code in the test harness, creating a circular dependency.

For now, I will create a separate crate which is just the rust code used by the "swift calling into rust" tests. Or is the circular dependency / two-way bridge scenario supposed to work?

Ok, this was a slight misdiagnosis of the problem, but I was close. I'm going to head to bed now, but I got pretty close to getting this working. I'll push a draft PR in a minute.

When running swift-on-the-server, you use Swift Package Manager (SPM) rather than Xcode to build. SPM uses Package.json to define packages, for example:

let package = Package(
    name: "IntegrationTests",
    targets: [
        // The compiled static rust library.
        .systemLibrary(
            name: "RustLib"),
        // The generated Swift wrapper code for the Rust library, plus some
        // Swift code used by the Rust library.
        .target(
            name: "SharedLib",
            dependencies: ["RustLib"]),
        .testTarget(
            name: "IntegrationTests",
            dependencies: ["SharedLib", "RustLib"]),
    ]
)

One big difference between xcode and SPM is that xcode allows shared-source targets, while SPM does not. From the docs:

Targets can contain Swift, Objective-C/C++, or C/C++ code, but an individual target can’t mix Swift with C-family languages. For example, a Swift package can have two targets, one that contains Objective-C, Objective-C++, and C code, and a second one that contains Swift code.

The current design of swift-bridge's generated output requires mixed-source packages; the package-name.swift generated file uses types from the package-name.h file. This can be worked around by generating two targets and importing one from the other, but this implies a coupling between the generation code and Package.swift, which is probably undesirable.