No reports generated...

[tomasharkema]

Hi, I really like the work you've done here. Very impressive. I hope reporting my issue can make this project ever more robust.

I've added several devices. Raspberry PI's via HCI and ESP32's via the deploy tool. All are working, and becoming green in the list, but for over two weeks there're still no reports found.

I must say that I'm not leaving home as much because of uncle rona, so maybe only my devices see my haystacks.

By compiling it from source, I see that some calls are erroring:

Error Domain=NSURLErrorDomain Code=-1012 "(null)" UserInfo={NSErrorFailingURLStringKey=https://gateway.icloud.com/acsnservice/fetch, NSUnderlyingError=0x600001b4c1b0 {Error Domain=kCFErrorDomainCFNetwork Code=-1012 "(null)" UserInfo={_kCFURLErrorAuthFailedResponseKey=<NSHTTPURLResponse: 0x6000014ea300> { URL: https://gateway.icloud.com/acsnservice/fetch } { Status Code: 401, Headers {
    "Apple-Originating-System" =     (
        UnknownOriginatingSystem
    );
    Connection =     (
        "keep-alive"
    );
    "Content-Length" =     (
        0
    );
    Date =     (
        "Mon, 26 Apr 2021 09:21:50 GMT"
    );
    Server =     (
    	[REDACTED]
    );
    "Strict-Transport-Security" =     (
    	[REDACTED]
    );
    Via =     (
    	[REDACTED]
    );
    "Www-Authenticate" =     (
    	[REDACTED]
    );
    "X-Apple-Edge-Response-Time" =     (
    	[REDACTED]
    );
    "X-Apple-Jingle-Correlation-Key" =     (
    	[REDACTED]
    );
    "X-Apple-Request-UUID" =     (
    	[REDACTED]
    );
    "X-Responding-Instance" =     (
    	[REDACTED]
    );
    "access-control-expose-headers" =     (
    	[REDACTED]
    );
    "apple-seq" =     (
    	[REDACTED]
    );
    "apple-tk" =     (
    	[REDACTED]
    );
} }}}, NSErrorFailingURLKey=https://gateway.icloud.com/acsnservice/fetch}
Failed with error dataCorrupted(Swift.DecodingError.Context(codingPath: [], debugDescription: "The given data was not valid JSON.", underlyingError: Optional(Error Domain=NSCocoaErrorDomain Code=3840 "No value." UserInfo={NSDebugDescription=No value.})))
Finished loading the reports. Now decrypt them
Decrypting reports

To Reproduce
Steps to reproduce the behavior:

1. Add new device
2. Deploy device and/or configure HCI
3. Mark as deployed
4. Wait....

Expected behavior
See reports after some time

Screenshots

OpenHaystack version:
0.3.6

macOS version:
11.2.3 (20D91)

[Sn0wfreezeDev]

Hi @tomasharkema,

Thank you for the detailed feedback. To me this looks like an error when retrieving the tokens necessary for downloading reports at Apple's servers. This is done by the Apple Mail Plugin, which you have installed correctly (green dot in the top right corner).
Are you using iCloud on your Mac? And is your Mac connected to an iCloud account? Because if this is not the case this might be the issue

[tomasharkema]

My Mac is indeed connected to an iCloud account. Even multiple for apple's 2FA. Is that maybe the issue? Happy to debug it some more if you have any direction.

Also the mail bundle connection appears to be working.

[Sn0wfreezeDev]

The function that is creating your access tokens is this one:

openhaystack/OpenHaystack/OpenHaystack/ReportsFetcher/ReportsFetcher.m

Line 121 in b5a577e

- (void)queryForHashes:(NSArray *)publicKeys

It needs a searchPartyToken, anisette data and your Apple user id.
Either of them might result in an error. So it's probably best to check all of them.

I can imagine that the Apple user id might be one from a different account in your case. Check the function fetchAppleAccountId to see if the correct one is returned. You can find the correct one by check Keychain access and searching for your Apple ID E-Mail. One entry should be an application password and it contains an account label with a number. This is you fix apple user id.

Then you can check if the search party token is actual data po searchPartyToken.

Then you might try to set a breakpoint here:

openhaystack/OpenHaystack/OpenHaystack/ReportsFetcher/ReportsFetcher.m

Line 143 in b5a577e

if (dict == nil) {

And print what's in the dict po dict (don't share it here)
The dict should contain at least:

{
    "X-Apple-I-MD" = 
    "X-Apple-I-MD-M" = 
}

It's quite a bit, but this should cover all areas where an error might arise

[tomasharkema]

Aha all data is correctly returned. But indeed, in fetchAppleAccountId I get the wrong Apple ID. It appears to be just a random one, not related to the sequence defined in Internet Accounts in System settings.

I'll look into how to differentiate the users primary account from keychain. Might be a real pain. Maybe match it with the appstore receipt? Might be that the user that's logged in into the App Store is the same as iCloud. Or maybe provide a picker in some onboarding, but that needs more rework.

[tomasharkema]

ah, got it working by implementing this:

- (NSString *)fetchAppleAccountId {
  NSDictionary *query = @{
      (NSString *)kSecClass : (NSString *)kSecClassGenericPassword,
      (NSString *)kSecAttrService : @"iCloud",
      (NSString *)kSecMatchLimit : (id)kSecMatchLimitAll,
      (NSString *)kSecReturnAttributes : @true
  };
  CFTypeRef items;
  OSStatus status = SecItemCopyMatching((__bridge CFDictionaryRef)query, &items);

  if (status == errSecSuccess) {
    NSArray<NSDictionary *> *itemsArray = (__bridge NSArray<NSDictionary *> *)(items);

    NSDictionary __block *itemDict;
    if ([itemsArray count] > 1) {
      NSLog(@"Multiple items!");

      [itemsArray enumerateObjectsUsingBlock:^(NSDictionary * _Nonnull obj, NSUInteger idx, BOOL * _Nonnull stop) {
        NSString *email = obj[(NSString *)kSecAttrLabel];
        NSLog(@"%@", email);
        if ([email hasPrefix:@"tomas@h"]) {
          itemDict = obj;
        }
      }];
    } else {
      itemDict = itemsArray[0];
    }

    NSString *accountId = itemDict[(NSString *)kSecAttrAccount];

    return accountId;
  }

  return nil;
}

(pretty shocked I still remember the enumerateObjectsUsingBlock block syntax)

so if ([email hasPrefix:@"tomas@h"]) { needs to be implemented by some way to get the main user email address. A quick google yielded what we maybe can get it via CloudKit.

Oh and by the way, I get results now!

[darkroastedOld]

Hey, im having the same issue with openhaystack. My devices do turn green (at the dot) but just like your issue it wont show a location.

Have you found an (easy to use) solution? And has the bug suggestion that you posted above been added to the release yet?

Thanks, any help is greatly appreciated!

[Sn0wfreezeDev]

Hi,

it could be that you suffer from the same issue. We have not fixed this in the latest release.
The suggestion by @tomasharkema is actually using his name in (which is a good fix for him). But this does not work for everyone, so we need to find another workaround.

[darkroastedOld]

Hey SnowFreeze, thanks for ur reply. Would there be a possibility for the devs to work this into the new release? If not could you maybe tell me what files to edit? And what to change?

Thanks

Edit: i have now got it working, thanks for the help

[linusheck]

Hey SnowFreeze, thanks for ur reply. Would there be a possibility for the devs to work this into the new release? If not could you maybe tell me what files to edit? And what to change?

Thanks

Edit: i have now got it working, thanks for the help

I have the same problem, how did you get it working?

I don't think I have multiple accounts linked - the code above just crashes and the array looks like there is only a single account.

[shadowHacker2020]

Hey SnowFreeze, thanks for ur reply. Would there be a possibility for the devs to work this into the new release? If not could you maybe tell me what files to edit? And what to change?
Thanks
Edit: i have now got it working, thanks for the help

I have the same problem, how did you get it working?

I don't think I have multiple accounts linked - the code above just crashes and the array looks like there is only a single account.

Yes dude , i need help as well

[linusheck]

Hey SnowFreeze, thanks for ur reply. Would there be a possibility for the devs to work this into the new release? If not could you maybe tell me what files to edit? And what to change?
Thanks
Edit: i have now got it working, thanks for the help

I have the same problem, how did you get it working?
I don't think I have multiple accounts linked - the code above just crashes and the array looks like there is only a single account.

Yes dude , i need help as well

For me it turned out that my device just didn't work, but it's really hard to debug because it's such a black box until locations appear.