funcptr - macOS

Mac OS X El Capitan Installer Removes Custom Group ID and Membership

2015-10-02T01:23:45+00:00

As always, after Apple releases their new operating system, my systems are upgraded. This time the upgrade was less of a surprise in terms of what it brings because I'd been beta testing the new release for the past couple of weeks, however I was still caught off guard.

On OS X, by default all user accounts start at ID 501 and count up, so if you have two accounts, you will have user ID 501 and 502 in use. For most people they will most likely never change this, and all is well. The default group ID for all new user accounts is staff</code> which has a group ID of 20. So if you have a single account named for example janedoe</code> her user ID would be 501 and her group ID would be 20 (staff</code>).

Coming from a FreeBSD world and running a lot of FreeBSD systems, user accounts start at 1001, and count up. When you create a new user account on FreeBSD, by default that user is also added to a group with the same name as the username, with the same ID. So you end up with an account with ID 1001 and default group ID 1001. Using the same example, a user named janedoe</code> would have a user ID of 1001, and a group ID of 1001 (janedoe</code>).

When I first installed OS X, and almost every single new installation since, I have followed these steps to change my user ID and group ID to match those on my FreeBSD systems:


Assumption is that you have a separate user account other than the one you
are about to modify that you can temporarily use that has administrator
privileges on the local Mac; I create an "Administrator" account for that
reason.</li>
System Preferences</li>
Users and Groups</li>
Click the +</code> (You may need to click the lock in the bottom left first)</li>
Change the dropdown to group</code></li>
Enter Full Name: janedoe</code></li>
Create group</li>
Right click on group (janedoe</code>)</li>
Advanced Options...</li>
Change the Group ID to 1001</li>
Okay</li>
Right click on user (janedoe</code>)</li>
Advanced Options...</li>
Change User ID from 501</code> to 1001</code></li>
Change Group from staff</code> to janedoe</code></li>
Okay</li>
Close System Preferences</li>
Open Terminal, become root user (sudo su)</li>
cd /Users/janedoe</code></li>
find . -uid 501 -print0 | xargs -0 chown 1001:1001</code></li>
</ol>
This allows me to have the same user ID and group ID on both my Mac OS X and on
FreeBSD, thereby making it easier to use tools like rsync that keeps ownership
and permissions, as well as using NFS. Other ways to do something similar is
using LDAP/Kerberos with shared directory service, but that is a little heavy
handed for a home network.</p>
This has worked for me without issues since OS X 10.8, even upgrading from 10.8 to
10.9 and then 10.10 did not change anything. However as soon as I did the
upgrade to El Capitan (10.11) I noticed that all of my ls -lah</code> output looked
like this:</p>
drwxr-xr-x+  13 xistence  1001   442B Oct  1 16:58 Desktop</span></span>
drwx------+  28 xistence  1001   952B Aug 31 12:17 Documents</span></span>
drwx------+  89 xistence  1001   3.0K Oct  1 15:56 Downloads</span></span>
drwx------@  72 xistence  1001   2.4K Oct  2 00:16 Library</span></span></code></pre>
and id</code> provided this valuable output:</p>
uid=1001(xistence) gid=20(xistence) groups=20(xistence),12(everyone),61(localaccounts),399(com.apple.access_ssh),402(com.apple.sharepoint.group.2),401(com.apple.sharepoint.group.1),100(_lpoperator)</span></span></code></pre>
Wait, what happened to the staff</code> group that I am supposed to be a member of,
and why is my xistence</code> group ID now stating it is 20 and not 1001 as I was
expecting.</p>
I wondered if the upgrade had messed up my group somehow, and it was
confirmed when I checked with dscl</code>.</p>
$ dscl . -read /Groups/xistence</span></span>
[...]</span></span>
Password: *</span></span>
PrimaryGroupID: 20</span></span>
RealName: xistence</span></span>
RecordName: xistence</span></span>
RecordType: dsRecTypeStandard:Groups</span></span></code></pre>
Do note that the group xistence</code> does not show up in System Preferences</em> ->
Users and Groups</em>, so we'll have to do some command line magic.</p>
Well, that's worrisome, why is this matching a built-in group's ID? Specifically
let's check the staff</code> group and make sure it still has the appropriate group
ID.</p>
$ dscl . -read /Groups/staff</span></span>
[...]</span></span>
GroupMembership: root</span></span>
Password: *</span></span>
PrimaryGroupID: 20</span></span>
RealName: Staff</span></span>
RecordName: staff BUILTIN\Users</span></span>
RecordType: dsRecTypeStandard:Groups</span></span></code></pre>
Next I had to check to see what my user account was set to as the default group
ID:</p>
$ dscl . -read /Users/xistence</span></span>
[...]</span></span>
NFSHomeDirectory: /Users/xistence</span></span>
Password: ********</span></span>
PrimaryGroupID: 20</span></span>
RealName:</span></span>
 Delta Regeer</span></span>
RecordName: xistence delta@regeer.org com.apple.idms.appleid.prd.53696d524c62372b48344a53755864634e4f374b32513d3d</span></span>
RecordType: dsRecTypeStandard:Users</span></span>
UniqueID: 1001</span></span>
UserShell: /bin/bash</span></span></code></pre>
Well, that is not entirely what I was expecting either, at last it didn't touch
my user ID. Time to fix things.</p>
First let's change the xistence</code> group's group ID to 1001, and then change the
Primary Group ID for the user xistence</code> to group ID 1001.</p>
dscl</span> .</span> -change</span> /Groups/xistence PrimaryGroupID</span> 20 1001</span></span>
dscl</span> .</span> -change</span> /Users/xistence PrimaryGroupID</span> 20 1001</span></span></code></pre>
After that id</code> looked a little bit more sane:</p>
uid=1001(xistence) gid=1001(xistence) groups=1001(xistence),12(everyone),61(localaccounts),399(com.apple.access_ssh),402(com.apple.sharepoint.group.2),401(com.apple.sharepoint.group.1),100(_lpoperator)</span></span></code></pre>
However now the group staff</code> is missing from the list of groups that the user
xistence</code> is a member of, which I don't think will hurt anything, but we still
want to be able to read/write any folders that are designated as staff</code>
elsewhere in the OS, and any other privileges that entails. So let's add the
user xistence</code> to the staff</code> group:</p>
dscl</span> .</span> -append</span> /Groups/staff GroupMembership xistence</span></span></code></pre>
Let's verify, and check id</code> again:</p>
uid=1001(xistence) gid=1001(xistence) groups=1001(xistence),12(everyone),20(staff),61(localaccounts),399(com.apple.access_ssh),402(com.apple.sharepoint.group.2),401(com.apple.sharepoint.group.1),100(_lpoperator)</span></span></code></pre>
For this to fully take effect, log out and log back in. This will make sure
that all new files have the correct user ID/group ID set.</p>
After the change to the Group ID, the group still doesn't show up in System
Preferences</em> -> Users and Groups</em>, which I find weird since it is not a
built-in group.</p>
Luckily everything is back to the way it was before the upgrade, and my backup
scripts and NFS shares work again without issues.</p>

IPv6 -- getaddrinfo() and bind() ordering with V6ONLY 2012-08-07T22:39:56+00:00 Recently I ran into an issue that took me a while to sort out, and it is regarding inconsistent behaviour on various OS's with regards to IPv6 sockets (AF_INET6</code></a>1</a>) and calling bind(2)</code></a> after getting the results back from getaddrinfo(3)</code></a>. A call to getaddrinfo()</code> with the hints set to AF_UNSPEC</code></a> in ai_family</code> and AI_PASSIVE</code></a> in ai_flags</code> will return to us 1 or more results that we can bind()</code> to. Sample code for that looks like this: struct addrinfo hints, *addrlist; memset(&hints, 0, sizeof(hints)); // Ask for TCP hints.ai_socktype = SOCK_STREAM; // Any family works for us ... hints.ai_family = AF_UNSPEC; // Set some hints hints.ai_flags = AI_PASSIVE | // We want to use this with bind AI_ADDRCONFIG; // Only return IPv4 or IPv6 if they are configured int rv; if ((rv = getaddrinfo(0, "7020", &hints, &addrlist)) != 0) { fprintf(stderr, "getaddrinfo: %s", gai_strerror(rv)); return 1; } // Use the list in *addrlist for (addr = addrlist; addr != 0; addr = addr->ai_next) { // use *addr as appropriate } // Clean up the memory from getaddrinfo() freeaddrinfo(addrlist);</code></pre> On Linux there are two entries returned when the host it is run on has both IPv4 and IPv6 enabled. An AF_INET</code> which was followed by an AF_INET6</code>. Now, it is not said that you are required to use all of the results that are returned, but if you want to listen on all address families it is off course suggested. Following the steps below for each of the returned results should result in having 1 or more different sockets that are bound to a single port. Create the socket()</li> Set any socket options you want (SO_REUSEADDR</code> for example)</li> Then bind() the socket</li> After that call listen() (followed off course by accept() on the socket)</li> </ol> Only for some unknown reason (and errno</code> is no help) bind()</code> fails when you get to the AF_INET6</code>, which was returned second. Searching online as to why the bind would fail doesn't give you any good results and the thing that is even worse is that if you run the same code on another platform such as FreeBSD</a>, OpenIndiana</a> or Mac OS X</a> no such failure exists. However I started suspecting something was up when I started looking at the output from netstat -lan | grep 7020</code> on Mac OS X. Where 7020 is the port I passed into getaddrinfo()</code>. tcp46 0 0 *.7020 *.* LISTEN tcp4 0 0 *.7020 *.* LISTEN </code></pre> Wait a minute ... one of the sockets is on both IPv4 and on IPv6. Some more time spent searching the internet I came across RFC 3493 section 5.3</a>, which is titled "IPV6_V6ONLY</code> option for AF_INET6</code> Sockets". As stated in section <3.7 Compatibility with IPv4 Nodes>, AF_INET6</code> sockets may be used for both IPv4 and IPv6 communications. Some applications may want to restrict their use of an AF_INET6</code> socket to IPv6 communications only. </blockquote> This was going down the right route, so I changed my code so that in the steps listed above in number 2 I added the following code if the socket type is AF_INET6</code>: if (setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, &yes, sizeof(int)) == -1) { close(sockfd); fprintf(stderr, "setsockopt: %s IPV6_V6ONLY\n", strerror(errno)); continue; }</code></pre> The RFC 3493 section 5.3 also states that this option should be turned off by default, which means that all IPv6 sockets can also communicate over IPv4. Thus technically setting the option manually in code the best way to fix the issue. FreeBSD has had this feature turned on (as in IPv6 sockets can only communicate with IPv6 and NOT IPv4) since 5.x. The biggest issue is that the remaining operating systems (OS X and OpenIndiana) don't have the same behaviour as Linux which makes troubleshooting this issue more difficult than it should be. The issue is that the RFC doesn't specify what exactly the operating should do when it encounters a request to bind to the same port on IPv4 and IPv6. The only place where I have found this documented is in "IPv6 Network Programming"</a> under "Tips in IPv6 Programming" chapter 4, section 4, appropriately titled "bind(2) Ordering and Conflicts". If you get a bind()</code> error when attempting to bind to an AF_INET6</code> socket please make sure that you set the socket option IPV6_V6ONLY</code> on the AF_INET6</code> socket. The default as required by RFC 3493 is to have that option be off. The default is wrong, and the RFC should have been more specific regarding what the right behaviour is when attempting to bind on an AF_INET6</code> socket when already bound on an AF_INET</code> while IPV6_V6ONLY</code> is set to false. The full code that I used for testing, along with a little bit more information is available as a gist on github</a>. The old BSD style socket()</code></a> called for defines starting with PF_</code> such as PF_INET</code> and PF_INET6</code> with the PF</code> standing for protocol family. POSIX starts them with AF_</code>, and calls them an address family. On almost every operating system PF_INET</code> is the same as AF_INET</code>. If the define doesn't exist you can always create it. ↩</a> </li> </ol> </section>