Dyld shared cache: Difference between revisions

From iPhone Development Wiki
(add dyld-cache-dump →‎Standalone tools)
 
(12 intermediate revisions by 2 users not shown)
Line 3: Line 3:
Since iPhone OS 3.1, all system (private and public) libraries have been combined into a big cache file to improve performance. The original files are redundant and thus eliminated from the system.
Since iPhone OS 3.1, all system (private and public) libraries have been combined into a big cache file to improve performance. The original files are redundant and thus eliminated from the system.


If you're looking for binaries or libraries inside of <tt>[http://theiphonewiki.com/wiki//System/Library/Frameworks /System/Library/Frameworks]</tt> or <tt>/System/Library/PrivateFrameworks</tt> (or other directories) and can't, this is why.
If you're looking for binaries or libraries inside of <tt>/System/Library/Frameworks</tt> or <tt>/System/Library/PrivateFrameworks</tt> (or other directories) and can't, this is why.


As of iOS 13.5 application code is now in frameworks in the shared cache. The binaries you see in <tt>/Applications</tt> or <tt>/private/var/staged_system_apps</tt> are now just shims, so if you attempt to class-dump them it will error out as no ObjC section will be found.
As of iOS 13.5 application code is now in frameworks in the shared cache. The binaries you see in <tt>/Applications</tt> or <tt>/private/var/staged_system_apps</tt> are now just shims, so if you attempt to class-dump them it will error out as no ObjC section will be found.
Line 11: Line 11:
== Obtaining a shared cache ==
== Obtaining a shared cache ==


=== From an iOS Device ===
=== For an iOS Device ===


==== Using the ipsw tool ====
==== Using the ipsw tool ====


# Install [https://github.com/blacktop/ipsw ipsw].
# Install [https://github.com/blacktop/ipsw ipsw].
# Run <code>ipsw download --version [target iOS version] --device [target device model (e.g. iPhone10,3)</code>
# Run <code>ipsw download ipsw --version [target iOS version] --device [target device model (e.g. iPhone10,3)</code>
# Run <code>ipsw extract --dyld</code>
# Run <code>ipsw extract --dyld</code>


Line 28: Line 28:
=== From a MacOS Device ===
=== From a MacOS Device ===


The MacOS dyld_shared_cache is located at <code>/usr/lib/dyld</code>
On newer machines, it is located at <code>/System/Library/dyld/</code>
 
Previously, it was located at <code>/usr/lib/dyld</code>


== Extracting Frameworks and Libraries. ==
== Extracting Frameworks and Libraries. ==
Line 53: Line 55:


Using [https://github.com/arandomdev/DyldExtractor DyldExtractor] is recommended for extraction. The only tool capable of extracting on its own is IDA Pro 7.5, and using an extractor can still be easier.
Using [https://github.com/arandomdev/DyldExtractor DyldExtractor] is recommended for extraction. The only tool capable of extracting on its own is IDA Pro 7.5, and using an extractor can still be easier.


==== IDA Pro ====
==== IDA Pro ====
Line 59: Line 62:


Details and instructions on using the integrated tools can be found on the [[IDA Pro]] page. If you own a copy, check the IDA page on this wiki for a guide on using it.
Details and instructions on using the integrated tools can be found on the [[IDA Pro]] page. If you own a copy, check the IDA page on this wiki for a guide on using it.


==== Hopper ====
==== Hopper ====
Line 64: Line 68:
Hopper is capable of loading singular modules from the shared cache. It does not fix references and as such produces mostly useless output.
Hopper is capable of loading singular modules from the shared cache. It does not fix references and as such produces mostly useless output.


It also allows loading the entire shared cache, but tends to crash when loading it. It's not worth your time.
It also allows loading the entire shared cache, but tends to crash when loading it.


==== Ghidra ====
==== Ghidra ====
Line 72: Line 76:
''More information is needed here''
''More information is needed here''


There are [https://github.com/NationalSecurityAgency/ghidra/issues/682 known issues] with loading a shared cache.  
There are [https://github.com/NationalSecurityAgency/ghidra/issues/682 known issues] with loading a shared cache.


=== Standalone tools ===
=== Standalone tools ===
Line 82: Line 86:
"Working":
"Working":


* [http://opensource.apple.com/source/dyld/ dsc_extractor (source code)]. More info [https://gist.github.com/NSExceptional/85527151eeec4b0640187a0a165da1cd here]. It produces the best results among all tools, but without branch islands workaround. Do note this is the exact same output provided by XCode automatically.
* [http://opensource.apple.com/source/dyld/ dsc_extractor (source code)]. More info [https://gist.github.com/NSExceptional/85527151eeec4b0640187a0a165da1cd here]. Do note this is the exact same output provided by XCode automatically.


Tools for previous iOS versions:
Tools for previous iOS versions:
Line 92: Line 96:
* [https://github.com/comex/imaon2 yasce] by comex is/was the best option for iOS 8 (and above), but you will need a nightly build version of rust; something like "rustc 1.9.0-nightly (339a409bf 2016-03-01)".
* [https://github.com/comex/imaon2 yasce] by comex is/was the best option for iOS 8 (and above), but you will need a nightly build version of rust; something like "rustc 1.9.0-nightly (339a409bf 2016-03-01)".
* [https://github.com/macmade/dyld_cache_extract dyld_cache_extract] by macmade that works on macOS and provides a complete GUI. Clone repo and do 'git submodule update --remote' before buidling. It was reported to be not working on iOS 10.2's dyld_shared_cache_armv7s; gave a 561.1MB executable file.
* [https://github.com/macmade/dyld_cache_extract dyld_cache_extract] by macmade that works on macOS and provides a complete GUI. Clone repo and do 'git submodule update --remote' before buidling. It was reported to be not working on iOS 10.2's dyld_shared_cache_armv7s; gave a 561.1MB executable file.
* [https://github.com/macmade/dyld-cache-dump dyld-cache-dump] by macmade: replacement for dyld_cache_extract<ref>https://github.com/macmade/dyld_cache_extract/issues/4</ref>


== Cache location ==
== Cache location ==
Line 166: Line 171:
</source>
</source>


== "Development" vs "Release" caches ==
== Class dumping ==
 
=== Creation/background ===
 
The source code for the tool used at apple is here:


https://opensource.apple.com/source/dyld/dyld-733.6/dyld3/shared-cache/CacheBuilder.cpp.auto.html
See [[Reverse_Engineering_Tools#Class/Metadata_Dumping_tools|this section of Reverse Engineering Tools]].
 
Specifically linked is the file responsible for determining if a development build is being created, and if so to avoid removing stubs. This is likely done to make debugging issues in frameworks much less painful.
 
Evidence of these builds is present in dyld source (''link to a github mirror here please''), and they're likely used in internal iOS builds.
 
== Class dumping ==


See [[Reverse_Engineering_Tools#class-dump.2C_class_dump_z.2C_classdump-dyld|this section of Reverse Engineering Tools]].
== External Links ==
== External Links ==


* [http://blog.howett.net/?p=75 Cache or Check?] — an analysis of the dyld_shared_cache system by D. Howett.
* [https://web.archive.org/web/20100609160737/http://blog.howett.net/?p=75 Cache or Check?] — an analysis of the dyld_shared_cache system by D. Howett.
* [https://github.com/deepinstinct/dsc_fix dsc_fix] — an IDA script that aids in reverse engineering dyld_shared_cache libraries
* [https://github.com/deepinstinct/dsc_fix dsc_fix] — an IDA script that aids in reverse engineering dyld_shared_cache libraries

Latest revision as of 14:55, 4 January 2024


Since iPhone OS 3.1, all system (private and public) libraries have been combined into a big cache file to improve performance. The original files are redundant and thus eliminated from the system.

If you're looking for binaries or libraries inside of /System/Library/Frameworks or /System/Library/PrivateFrameworks (or other directories) and can't, this is why.

As of iOS 13.5 application code is now in frameworks in the shared cache. The binaries you see in /Applications or /private/var/staged_system_apps are now just shims, so if you attempt to class-dump them it will error out as no ObjC section will be found.

OS X, along with any other *OS released by apple also uses a shared cache. Unlike iOS, macOS before 11 Big Sur used to ship with the source binaries still on-disk, particularly so it can be updated with update_dyld_shared_cache. Starting with macOS 11, update_dyld_shared_cache is deprecated and, as in iOS, the only copy of the libraries is in the "cache". The cache is only vaguely documented in dyld man pages.

Obtaining a shared cache

For an iOS Device

Using the ipsw tool

  1. Install ipsw.
  2. Run ipsw download ipsw --version [target iOS version] --device [target device model (e.g. iPhone10,3)
  3. Run ipsw extract --dyld

Manually

  1. Download or locate an .ipsw file for the target version and device
  2. Rename it to a .zip, extract, and locate the largest .dmg
  3. Mount the dmg and navigate to /System/Library/Caches/com.apple.dyld/
  4. Copy the dyld_shared_cache to your machine.

From a MacOS Device

On newer machines, it is located at /System/Library/dyld/

Previously, it was located at /usr/lib/dyld

Extracting Frameworks and Libraries.

Since iOS 8, the SDK no longer includes extracted frameworks. The only way to obtain the libraries running on your device is via extraction.

Extracting the shared cache is useful in a few situations:

  • Linking against a framework or library not available in the public SDK
  • Using class-dump or similar tools to analyze Private Frameworks.
  • Reverse engineering the code in private or public frameworks within iOS.

DyldExtractor is currently (as of iOS 14) capable of extracting nearly perfect frameworks from the shared cache. However, these cannot yet be loaded by dyld, as more work needs done.

Shared cache static analysis

Simulator Runtime Frameworks

Can someone with an M1 mac comment on this?

The simulator runtime generated by Xcode includes fully symbolicated, perfect x64 binaries for Private and Public frameworks. For most shared_cache analysis, unless you own IDA 7.5 or the frameworks you are looking for are not available in simulators, you are fine with these. They're located under /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot (Xcode 11 and higher) or /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot (Xcode 10 and lower).

Disassemblers capable of extraction

Using DyldExtractor is recommended for extraction. The only tool capable of extracting on its own is IDA Pro 7.5, and using an extractor can still be easier.


IDA Pro

IDA 7.5 includes a plethora of tools that make working with the dsc much more tolerable. It entirely eliminates the need for extraction or third-party scripts to fix output.

Details and instructions on using the integrated tools can be found on the IDA Pro page. If you own a copy, check the IDA page on this wiki for a guide on using it.


Hopper

Hopper is capable of loading singular modules from the shared cache. It does not fix references and as such produces mostly useless output.

It also allows loading the entire shared cache, but tends to crash when loading it.

Ghidra

Ghidra is capable of loading the shared cache

More information is needed here

There are known issues with loading a shared cache.

Standalone tools

Working on current iOS versions:

  • DyldExtractor. Python project capable of extracting and heavily fixing up frameworks from the shared cache. This tool produces nearly perfect output on current iOS versions.

"Working":

Tools for previous iOS versions:

  • dyld_decache by KennyTM~ to extract these dylibs.
  • DySlim by comex to mount the whole cache file on Mac OS X.
  • decache by phoenixdev to nearly perfectly extract dylibs from iOS <= 6 cache file.
  • jtool is another option starting from iOS 8.
  • yasce by comex is/was the best option for iOS 8 (and above), but you will need a nightly build version of rust; something like "rustc 1.9.0-nightly (339a409bf 2016-03-01)".
  • dyld_cache_extract by macmade that works on macOS and provides a complete GUI. Clone repo and do 'git submodule update --remote' before buidling. It was reported to be not working on iOS 10.2's dyld_shared_cache_armv7s; gave a 561.1MB executable file.
  • dyld-cache-dump by macmade: replacement for dyld_cache_extract[1]

Cache location

The cache is located in /System/Library/Caches/com.apple.dyld/dyld_shared_cache_armX, where X can be:

X Device ARM Architecture
v6 ARMv6
v7 ARMv7
v7s
v7k
64 ARMv8
64e ARMv8.3


Other Disassemblers

Information on how other disassemblers interact with these caches is needed.


Example usage for jtool

To extract a specific binary from the cache ("UIKit" can be replaced with a different framework or library):

jtool -extract UIKit path/to/dyld_shared_cache

An example of one way to dump all the binaries at once (be careful with this, it creates huge files):

cache=dyld_shared_cache_arm64
mkdir -p extracted && jtool -lv $cache | cut -c 24- | tail +5 | while read line; do mkdir -p extracted/"$(dirname "$line")"; jtool -extract $line $cache; mv $cache."$(basename "$line")" extracted/$line; done

Problems with jtool

Please be aware that decache produces currently (16.04.15) better and more usable results then jtool, as jtool fails to resolve and fix the "uniqued" objectiv c selectors correctly.

Apple "uniques" objectiv c selectors, such as "alloc" (alloc is used almost everywhere), which are used in more then one place, into a single one. When extracting an image from the cache, the address of such a shared selector will most likely not be in the extracted image anymore, so this needs to fixed, which jtool apparently fails to do. (For more information: http://opensource.apple.com/source/dyld/dyld-132.13/launch-cache/update_dyld_shared_cache.cpp, look at the class ObjCSelectorUniquer)

Not working since iOS 11

jtool2 is the newer version this user reports it not working on the iOS 11 shared cache - shows a warning "File is likely truncated (or header corrupt?)" and then doesn't get past "LC_DYLD_INFO..." http://newosxbook.com/forum/viewtopic.php?f=3&t=19577&start=50#p24183 Still same error exists on the cache from iOS 13.

$ ./jtool2 -e Stocks ./dyld_shared_cache_arm64 
Warning: File is likely truncated (or header corrupt?) Binding opcodes falls outside file
Warning: File is likely truncated (or header corrupt?) Binding opcodes falls outside file
Warning: File is likely truncated (or header corrupt?) LC_FUNCTION_STARTS falls outside file
0x176d9000-0x17731000 __TEXT   (360448 bytes)
0x2ccc2070-0x2ccd54a8 __DATA_CONST   (78904 bytes)
0x30571f60-0x30575f60 __DATA   (16384 bytes)
0x314c9a28-0x314ca000 __DATA_DIRTY   (1496 bytes)
0x316b3000-0x37064000 __LINKEDIT   (94048256 bytes)
   0x31e3c4d0-0x31e3eaf8 Exports   (9768 bytes)
   0x34689f18-0x346912f8 Symbol Table   (29664 bytes)
   0x35a46500-0x35a46b90 Function Starts   (1680 bytes)
   0x35bb1930-0x35bb1960 Data In Code   (48 bytes)
   0x35d66710-0x37029aba String Table   (19674026 bytes)
LC_DYLD_INFO...

Class dumping

See this section of Reverse Engineering Tools.

External Links

  • Cache or Check? — an analysis of the dyld_shared_cache system by D. Howett.
  • dsc_fix — an IDA script that aids in reverse engineering dyld_shared_cache libraries