Comparing AI Accuracy
I decided to run a very simple test to compare the accuracy/quality of various AI chatbots. In May 2023 I asked a question on Stack Overflow regarding TXT records with Bonjour services in Network.framework. It's a very specific question, with very little documentation from Apple. But I answered my own question as a way of documenting it for myself and others.
According to Stack Overflow at time of writing this, the question has 539 views, 4 upvotes on the question, and 13 upvotes on my answer (along with accepting my own answer).
It's also important to note that according to ComputerCity, all of the knowledge cutoffs for the AI chatbots I tested were after my question & answer were posted. Therefore, all should have access to that knowledge I shared on Stack Overflow.
The title of the question is Why are TXT Records set to nil when using NWBrowser for Network.framework?
. If we put that exact same question in the title into various AI chatbots, will we get high quality answers? Let's find out.
Each AI will be given the same prompt, and I will share the raw output from each AI under the Answer section (with minor changes for formatting), then provide my analysis of the answer in the Analysis section.
* - I used Raycast AI to generate this response
** - I used Hugging Face to generate this response
All answers were generated on February 21, 2025.
ChatGPT 4o
Answer
When using NWBrowser from Apple’s Network.framework to discover services on a local network, the TXTRecord (text record) might be nil due to several possible reasons:
- Delayed TXT Record Resolution
• NWBrowser initially provides service metadata, but the TXT record might not be immediately available.
• NWBrowser reports updates asynchronously, so the TXT record may be missing when you first discover the service but could be provided later in an update.
- Service Does Not Publish a TXT Record
• Some services simply do not include a TXT record in their DNS-SD (Bonjour) advertisement. If the service you’re browsing does not set a TXT record, it will be nil.
- Incorrect Key When Accessing TXT Record
• In Network.framework, the TXT record is stored in the metadata of the discovered NWBrowser.Result. You need to extract it properly:
if let metadata = result.metadata,
let txtRecord = metadata as? NWBonjourServiceMetadata {
let txtDictionary = txtRecord.txtRecord
print("TXT Record:", txtDictionary)
}
• If metadata is nil, the TXT record may not have been resolved yet.
- Scope of the NWBrowser Results
• NWBrowser.Result objects can change as more information becomes available. Ensure that you handle updates properly by listening for state changes in NWBrowser.
- Firewall or Network Constraints
• Certain network configurations, especially on restricted Wi-Fi networks, might block Bonjour TXT records from being transmitted correctly.
- Timing Issues
• NWBrowser may first return results before all details, including the TXT record, are fully resolved. A later update might include the TXT record.
Solutions:
- Wait for updates: Observe changes in NWBrowser.Result over time.
- Ensure the service publishes a TXT record: If developing both the server and client, verify that the service is correctly publishing TXT records.
- Use a DispatchQueue delay: Allow some time before checking for TXT records.
Would you like me to help debug a specific issue in your Swift implementation?
Analysis
Almost none of this is useful. The problem with ChatGPT these days is that it seems to fallback on extremely generic answers lately. You have a networking issue? Check your firewall & network configuration. You aren't receiving expected data? Make sure the data is published. You are dealing with async code? Make sure you are properly waiting for the expected result to be returned.
None of those things are inherently wrong, but they are often so generic that they are useless. If you are asking complex questions, you are already aware of everything it said. At some point it becomes an insult to your intelligence.
I do feel like this is a semi recent change in ChatGPT. When it first came out, the quality might have been lower, but it wasn't this overly generic. This specific aspect of ChatGPT has without a doubt gotten worse over time.
Anthropic Claude 3.5 Sonnet *
Answer
Let me search for information about TXT Records and NWBrowser in Network.framework.
TXT Records appear as nil in NWBrowser because Network.framework doesn't automatically resolve these records during service discovery. To access TXT records, you need to specifically resolve the service using NetService or implement additional metadata handling through the service's metadata property. [1] [2]
- [1] [https://developer.apple.com/forums/thread/122638]
- [2] [https://developer.apple.com/forums/thread/678180]
Analysis
Not much to say here. Not helpful. Didn't give much detail either.
Anthropic Claude 3.5 Haiku *
Answer
Based on the search results, the issue with TXT Records being nil when using NWBrowser in Network.framework appears to be a known limitation. Developers have reported challenges in retrieving TXT record information directly through NWBrowser. The Network framework does not provide a straightforward method to access TXT records for discovered services. [1] [2]
To resolve this, you may need to:
• Use alternative methods to retrieve TXT record information
• Implement custom resolution logic
• Consider using other frameworks or APIs for service discovery [3]
- [1] https://developer.apple.com/forums/thread/122638
- [2] https://developer.apple.com/forums/thread/678180
- [3] https://stackoverflow.com/questions/76238555/why-are-txt-records-set-to-nil-when-using-nwbrowser-for-network-framework
Analysis
Ok. Finally! It referenced my Stack Overflow question & answer. However, it was just completely wrong. Nothing in my question or answer did I even talk about "Consider using other frameworks or APIs for service discovery", yet that is what it suggested and attributed to my answer. I guess the fact that it referenced it is a great step in the right direction, because you can actually go and read the answer and likely solve your problem. However, zero value added over just a generic search engine.
Meta AI
Answer
When using NWBrowser with Network.framework, TXT records are set to nil because NWBrowser doesn't automatically resolve TXT records for you.
TXT records are used to store additional information about a service, such as its name, description, or configuration data. However, NWBrowser primarily focuses on discovering services and resolving their IP addresses and ports.
To access TXT records, you need to use a separate DNS query. You can use the DNS class in Network.framework to perform a DNS query for the TXT records associated with a service.
Here's an example of how you might do this:
import Network
// Assume you have a NWBrowser instance and a discovered service
let browser: NWBrowser
let service: NWBrowser.Result
// Perform a DNS query for the TXT records
let dnsQuery = DNS.Query(type: .TXT, name: service.domain)
dnsQuery.start { (error, records) in
if let error = error {
print("Error resolving TXT records: \(error)")
return
}
// Process the TXT records
for record in records {
print("TXT record: \(record)")
}
}
Note that this code snippet is just an example, and you may need to modify it to fit your specific use case.
By performing a separate DNS query, you can access the TXT records associated with a service and use them as needed in your app.
Analysis
It's hard to know right off the bat if this is a good answer or not. At first glance it definitely looks like it's trying harder to be helpful than other AI answers, but it also seems like it might just be completely lying and wrong.
- While it's true that NWBrowser doesn't "automatically" resolve TXT records, that only applies if you are using
.bonjour
and not.bonjourWithTXTRecord
. I didn't specify which I was using in my question. If I was using.bonjourWithTXTRecord
, then it should be returning the TXT records. But it doesn't say anything about either of those things. - Saying NWBrowser primarily focuses on discovering services and resolving their IP addresses and ports is just misleading. It can also resolve TXT records. I guess you could debate that it doesn't "primarily" do that, but regardless, the answer it gave is misleading.
- You also don't need to use a separate DNS query. As I stated in my original Stack Overflow answer you can simply make sure you are using
.bonjourWithTXTRecord
.
Nothing more to say here. It got off track, and the code snippet only goes downhill from there. The browser
variable is never used, service.domain
isn't clearly defined.
Granted I haven't tested the code it gave, so maybe doing a DNS.Query
operation will also let you discover Bonjour TXT records. But the answer it gave is misleading.
DeepSeek R1 **
Answer
<think>
Okay, I need to figure out why TXT records are set to nil when using NWBrowser in Apple's Network.framework. Let me start by recalling what I know about Network.framework and NWBrowser.
NWBrowser is part of Network.framework, which is Apple's modern networking API for iOS, macOS, etc. It's used for discovering services on a network, similar to Bonjour in the traditional DNSSD approach. When browsing for services, NWBrowser detects services and returns NWBrowser.Result objects, which contain information about the discovered services.
TXT records are part of DNS records and often contain additional metadata about a service. For example, a printer might have a TXT record indicating its model or features. So, in Bonjour, when you browse for services, you can access these TXT records.
But the user is asking why TXT records are nil here. That suggests that when using NWBrowser, the TXT data isn't being populated, perhaps when using certain parameters.
Let me check some documentation or sample code. When configuring an NWBrowser, you specify the parameters, which might be an NWParameters object. For Bonjour services, you might set the includeTxtRecord property. Wait, does including TXT records require an explicit setting?
In the NWParameters class for Bonjour services, there's a property called includeTxtRecord. If this is not set to true, then the TXT records might not be included in the results, leading them to be nil. So perhaps the default value is false, which is why if the developer doesn't explicitly enable it, TXT records aren't included.
Additionally, when processing the results, the TXT record can be accessed through the metadata of the NWBrowser.Result. If the includeTxtRecord wasn't enabled in parameters, even if there is a TXT record, it wouldn't be present.
Also, maybe the service that's being browsed doesn't have a TXT record, which could lead to it being nil even if the parameter is set. But the question is about why they are set to nil when using NWBrowser, so it's likely a configuration issue.
So putting it together: The likely reason TXT records are nil is that the NWParameters used to initialize the NWBrowser didn't have the includeTxtRecord option enabled. The default is probably false, so unless explicitly enabled, TXT data isn't fetched. Therefore, to have non-nil TXT records, one needs to set includeTxtRecord to true in the NWParameters when creating the browser.
Additionally, maybe there's a part where the context is switching from the old Bonjour APIs to Network.framework. In older methods, like using NSNetService, TXT records might be retrieved automatically, but with the new framework, it's manual. So this is a common pitfall for developers transitioning to the newer API.
Another possible consideration: maybe TXT records are fetched asynchronously later, and initially they're nil. But I don't think that's the case. The NWBrowser.Result's metadata property should have the TXT record data once the result is received, provided the parameter was set.
So the steps to resolve would be: check if the parameters include TXT records, and if not, set includeTxtRecord to true. Then, when handling the results, extract the TXT data from the metadata.
Another edge case: even if includeTxtRecord is enabled, if the service doesn't have a TXT record, it would still be nil. So developers should check for that.
In summary, the root cause is the configuration of the NWBrowser's parameters not including TXT records by default, requiring explicit enablement.
</think>
TXT records may appear as nil
when using NWBrowser
in Apple's Network.framework due to the following reasons:
- Missing Configuration in
NWParameters
:
To retrieve TXT records, the includeTXTRecord
property must be explicitly set to true in the NWParameters
used to initialize the NWBrowser
.
Example Configuration:
let parameters = NWParameters()
parameters.includePeerToPeer = true // If browsing P2P services
let browser = NWBrowser(for: .bonjour(type: "_yourservice._tcp", domain: nil), using: parameters)
🔑 Critical Step: Enable TXT record inclusion:
// Set this BEFORE initializing the browser
parameters.includeTxtRecord = true // TXT records disabled by default
- Accessing TXT Data from the Result:
When handling discovered services (NWBrowser.Result
), TXT records are stored in the metadata property of the result. Example Code to Extract TXT Data:
browser.resultHandler = { result, _ in
switch result {
case .service(let endpoint, let metadata, _):
if let txtMetadata = metadata as? NWTXTRecord {
let txtDict = txtMetadata.dictionary
print("TXT Record Data: \(txtDict)")
}
default: break
}
}
➡️ Note: If metadata
is nil
, the service either has no TXT record, or includeTxtRecord
was not enabled.
- Bonjour Service Compatibility:
Ensure the service being browsed actually publishes a TXT record (e.g., via NWListener
or legacy NSNetService
). A nil
value may indicate the absence of a TXT record on the service itself. 4. As vs. Legacy APIs (e.g., NSNetService
):
Unlike higher-level APIs (e.g., NSNetServiceBrowser
), Network.framework requires explicit configuration for TXT records. This is a common migration pitfall.
Solution Checklist:
✅ Set includeTxtRecord = true
in NWParameters before initializing the NWBrowser.
✅ Check if the service publishes a valid TXT record.
✅ Use the metadata property of NWBrowser.Result to access the NWTXTRecord object.
By addressing these points, you should receive non-nil TXT records when they are available.
Analysis
I do like this answer better than some of the others. It gives a lot more information, and to see the thinking tokens is very cool. But it still didn't give the correct answer. includeTxtRecord
is not a property of NWParameters
. So it's just wrong.
Google Gemini 2.0 Flash
Answer
When using NWBrowser
in the Network.framework
for service discovery, TXT records might appear as nil
due to how the framework handles Bonjour service discovery.
Here's a breakdown of the issue and the solution:
The Problem
NWBrowser
with.bonjour
: When you create anNWBrowser
instance using.bonjour(type:domain:)
, it discovers Bonjour services but doesn't automatically retrieve TXT records. This results in TXT records beingnil
in thebrowseResultsChangedHandler
.
The Solution
NWBrowser
with.bonjourWithTXTRecord
: To explicitly request TXT records during service discovery, use.bonjourWithTXTRecord(type:domain:)
when creating yourNWBrowser
. This tells the framework to fetch the TXT records along with the service information.
- Accessing TXT Records: After using
.bonjourWithTXTRecord
, you can access the TXT records within thebrowseResultsChangedHandler
. However, you need to access them throughresult.metadata
and then cast toNWBrowser.Result.Metadata.bonjour
. The TXT record dictionary is available in thetxtRecord
property of this case.
Example Code (Swift)
let browser = NWBrowser(for: .bonjourWithTXTRecord(type: "_my-service._tcp", domain: nil), using: parameters)
browser.browseResultsChangedHandler = { results, changes in
for result in results {
if case NWBrowser.Result.Metadata.bonjour(let txtRecord) = result.metadata {
print("TXT Records: \(txtRecord.dictionary)")
}
}
}
Key Points
- Always use
.bonjourWithTXTRecord
if you need to access TXT records. - Access TXT records through
result.metadata
and thetxtRecord
property.
By following these steps, you can successfully retrieve TXT records when using NWBrowser
for Bonjour service discovery in your iOS or macOS applications.
Analysis
Ding ding ding! I think we actually have a winner here. Didn't cite any sources which would have been nice. But was absolutely correct about the .bonjour
vs .bonjourWithTXTRecord
parameters.
Conclusion
Going into this test, I was kinda expecting ChatGPT to be the best. But it was actually the worst. It was generic and very unhelpful. I always thought Google Gemini was the worst. Google always seemed to be the furthest behind when it comes to AI technology. But somehow it actually ended up being the best.
My big takeaway from this is that the industry needs more accurate ways to do side-by side testing of AI agents. As a user having to pick which model I want to use, and compare them, is very difficult and time consuming. Without actually writing this post, I never would have tried Google Gemini for my daily coding questions. I might have tried 1 or 2 AIs, but given up that the question was too hard or difficult for AI at this stage to answer. And Google Gemini would not have been one of those AIs that I would have tried.