Beats Off v0.1

Note: Despite the screenshot, Beats Off (if you have a better name, please let me know) is cross-platform!
Windows and Linux binaries included in the download.
I need to make more VAGs.

After a brief crash-course in REALbasic (I probably should have used something better, but it has easy GUI support, is cross-platform, and has easy file I/O functions) I've made a simple application that will allow you to organize VAG files into tracks and pages. The app will then build a working BJS file and minimal BAR for use in Beats.

Limitations (in order of severity/annoyance) include:

• The BAR file created must replace an already existing Jam BAR.
  This is due to a single damn value in the header that I can't figure out how to generate. JAM_0002 is the victim right now, though I quite like this Jam, so I might change it later.
  
• User needs to convert audio files to VAGs themselves.
  I'm not about to get into the business of file conversion, so I recommend Awave Audio for batch converting the dozens of files you'll need to make a decent Jam. Once this is done, however, my application includes a tool for batch modifying the VAG headers for use in Beats.
  
• User must know how to unpack, modify and repack the Beats ISO.
  If you've purchased Beats (please do), you can convert it to an ISO using NP Decryptor. From here you must extract the contents of the ISO, replace the proper folder, then repack the ISO and copy it over to your PSP. I use Prometeus for OS X to do the unpacking/repacking, but I'm sure there are other apps for other platforms.
• All loops must be 4 bars long.
  Beats allows for use of various sized loops, but right now it's locked in at 4. The duration of the main loop and time signature are also locked in at 4. This is an easy enough fix, I just need to create a better UI.
  
• XML is not generated yet.
  More laziness on my part: While the names created in the BJS will show up in Jamming mode, they will not show up in the menu; there will be a blank (but selectable) entry.
• No samples yet.
  At first I didn't think much of the samples in Beats, as they seemed to lag when you triggered them (either that or I have less rhythm than I thought). A little more complicated to add, but might be worth it for triggered effects.
• No cover art yet.
  The image format used by Beats is slightly different than what's normally used by, say, the XMB. My few attempts to create a custom image have failed, and at the moment this isn't my highest priority.
• I'm not a programmer.
  Other than the mess of amateur code under the hood (source available soon), there are probably some bugs, as I haven't thoroughly tested this. In fact, the Linux build is completely untested.

With that said, use at your own risk.
I'm not responsible for any problems that may arise from using this program.

See readme.txt for instructions, as the UI isn't all that intuitive.
Good luck, and enjoy!

Success!

Update: I was able to get a file to load and play! I was just sure to include all the values outlined in the BIFF. From what I can see, the chunk of data that follows a VAG file in the BJS is optional, as long as I set its length to 0. I just hope this doesn't break anything in the future.

On another note, FreePlay and I have been desperately trying to figure out the unknown values found in the header of BAR files. From what it looks like, they're related to the file name, as the manifests always have the same value, but the others change per the name of the folder. If we figure out out to encode our own names, it'll be possible to add content to Beats as opposed to replacing it.
I hand-built a really simple BJS with only one loop in it. It loads, counts off at the right tempo... then freezes. I didn't add the unknown data that would normally follow the VAG, so that might just have something to do with it.
Interestingly, I couldn't name my folder JAM_0000. Renaming it to JAM_0002 works, but means one has to replace a song.

Be sure to check out FreePlay's excellent "unbeat" tool for extracting BAR files. A lot cleaner and faster than using File Juicer or doing it by hand.

Introductions

A few months ago when I purchased Beats, I thought it was a fun little game, and well worth the $5. Soon enough, however, I became bored with my music and the game's too easy/too hard settings and stopped playing.

But one day I started to play with oft-neglected "Jamming" mode and realized what a great portable DJ tool this could be. That is, if I could add my own loops and samples.

So, since April 26th, I've been working on reverse engineering the game, hoping to do just that. I think things are clear enough now to share my findings and hope the community can help me fill in a few holes and create a script to automate the (what seems to be) overly complicated process of building files Beats can use.

And no complaints about the URL, it's late and I thought it was funny.

Directory Structure

Despite purchasing Beats, I've been working on an ISO version of the game for simplicity. The structure is that of a typical PSP game.

In SYSDIR there's a dummy BOOT.BIN, working EBOOT.BIN and update files (I like to delete those). Decrypting EBOOT.BIN with PRXdecrypter yields a MIPS ELF binary with a lot of data attached to it, some in the form of the BAR file I will cover next. I didn't dive into it too much--as the Jamming data is elsewhere--but it should be noted that Beats was written in Java of all things. If that sparks anyone's interest, Beats might be worth looking into for that alone.

In USRDIR there's config.xml which isn't too useful, but there are more references to Java including the applet it loads on boot. I like to change this to "applet/com.scee.london.beats.frontend.Frontend" as it forces the game to skip the intro logos. Be forewarned, this also skips the automatic loading of your profile; all the settings will be set to default and Beats won't save any of your progress should you decide to play a quick game. USRDIR also contains a MODULES directory with font, MP3 and PSMF (video) libraries.

Most importantly, USRDIR contains a directory titled BARS. It is here where all the important data is stored. The directories within are named with a three letter ID and a number: CLG_100x are the inbuilt songs, JAM_000x is our Jamming data, THM_000x are themes, TUT_0000 is the tutorial and VIS_10xx are visualizers. (Deleting all these is a good way to save space and speed up ISO build time.) Within these directories are single files named BULK.BAR.

BAR Files

The data files used by Beats (BULK.BAR) are in a proprietary format that contain a number of zlib deflated files topped with a moderate header (not the simplest, but nothing like you'll see later). I first discovered the zlib'd files when I used the OS X program File Juicer in an attempt to find files embedded within the BAR. Jaeder Naub is the closest equivalent I could find for Windows, but it has troubles extracting the zlib'd files properly.

Before I start posting code it should be noted that any values posted are from the BULK.BAR file of JAM_0002. BAR files vary greatly across types and even within the same type, but the basic structure remains the same.

The header is as follows:

x00| E1 17 EF AD 00 00 00 01 Magic Number
x10| 05 00 00 00 Number of files in the BAR
This is important, as the size of the header depends on this.
Also the value is byte flipped or little endian. You'll see this A LOT.

From here on out there will be 16 byte chunks for each file. The order of these chunks isn't important.
I'll use the first chunk as an example:

x14| 5F AE 8F D1 Unknown related to zlib
I want to say it's a CRC, but I can't recreate it and all the JAM_000x BARs have the same value here. If it is a CRC, Beats semi-ignores it anyways; zeroing it causes failure, but leaving it despite a huge change in the file works OK.
x18| 00 00 00 00 Offset (minus header) of zlib
x1C| 03 01 00 00 Inflated size of zlib (259)*
x20| C4 00 00 00 Deflated size of zlib (196)*

And repeat for each file.

After each file is represented in the header, the first file appears , beginning with 78 DA.

With this header info, one should have no problem (except maybe time) extracting the files for inflation.
I'm focusing on the Jamming files, but feel free to explore others. The visualizers contain a lot of XML and difficult GIM/MIG images. Themes have a bit of XML and PSMF background videos. I haven't really looked at the CLGs (songs) as there's already a method to add your own music. If you decrypt EBOOT.BIN you'll find an embedded BAR file. Be warned: there are A LOT of files in there, though this is where you'll find Java .class files; sounds, UI elements, and fonts; and even more XML.

*Inflating and deflating the files can be a pain as there are few tools that use the same algorithm. zlibc for Windows does both perfectly. If one was so inclined, it should be trivial to make a tool in Python for other platforms.

Files within a JAM_000x BAR

I'll continue to focus on Jamming files here, as this is my goal. Maybe in the future I'll get around to playing with the others, but first thing's first.

Following the header is a zlib deflated manifest file, duplicated here:

; Manifest: dance2

MINILOCALE
data/minilocale/dance2.xml, MS

TRACK_SELECTION_MS
title/dance2, title/jam_0002, texture/cover_dance2, dan, applet/com.scee.london.beats.jamming.Jamlet, data/jamming/dance2.bjs, freeplay|transferable, easy, all_players

Now I know absolutely no Java, so I'm probably wrong, but this doesn't look like a normal Java manifest. Any case, stripping this file out of the BAR has no known effects. The only thing I could see it doing is separating easy, medium, and hard songs; preventing wifi sharing, or limiting its availability to certain players in a hypothetical unlock-the-song-style game.

Next file in the BAR is the aforementioned XML file called "MINILOCALE." This changes the obscure short names within a BJS file to user-friendly strings per the user's language:

<minilocale> //File type
<en_en> //Language
<roles> //Pages or instruments
<dan1>DRUMS</dan1> //Follows <bjsname>STRING format
<dan2>BASS</dan2> //where danx is the page name found in the BJS
<dan3>SYNTHS</dan3> //and STRING is what's displayed on screen
<dan4>FX</dan4>
</roles>
<title>
<dance2>SMACKDOWN</dance2> //Song title string using internal name as tag
<jam_0002>SWITCHSHIFT</jam_0002> //Artist name using folder name as tag
</title>
<tracks>
<jam_0002_t1>DRUMS 1</jam_0002_t1> //Uses names found within the BJS as tags
<jam_0002_t2>DRUMS 2</jam_0002_t2>
...
<jam_0002_tc>FX 3</jam_0002_tc>
</tracks>
<samples>
<jam_0002_s1>PERCUS.</jam_0002_s1> //Sample name (one-time use sounds you can use to drum along in the song.)
</samples> //This song in particular actually doesn't have any samples; this is just an artifact.
</en_en>
And on, for five more languages: Fr, Es, It, De, and Jp.

Next is the cover art file. This is a 8 bit 128x128 PSP MIG file similar to the ones used for the PSP and PS3 GUIs. There are a couple of utilities that will convert to/from MIG files, but I've been unable to create a custom file that will display in the game. It might be due to some custom header, but I'm not too familiar with the format and haven't had a chance to look into it, as my focus has been on the audio portion.

The above two files are only for the UI, as I've had no problem loading a BAR with everything but the BJS stripped out. If you do that, there's a nice blank spot in the song list, but it loads and runs fine, and uses the internal "dan1" and "jam_0002_t1" names for the UI instead.

The fourth file in the BAR is the all-important BJS file which I'll cover in the next post.

And the final file is actually another BAR file that contains a smaller manifest and a duplicate of the XML seen above. This seems to be an artifact, as I'm able to remove it with no known consequences, though it might be used for Ad Hoc play. I can't test this as I have only one PSP, but wireless Jamming does look pretty fun; each PSP is assigned an instrument. If one was set up with 4 PSPs, they wouldn't have to worry about switching pages while performing, fixing the slight flaw of being unable to move backwards through the pages.
Oh if only the PSP had a touch screen.

VAG Files

Before I delve into the complicated structure of the BJS, let's look at the sound files it contains:
Loops and samples are stored as Sony ADPCM VAG files with byte flipped header values. They're always 16 bit mono at 44,100kHz, but Beats might support other formats. The best apps I've seen for creating VAG files are MFAudio and Awave Audio. MFAudio is freeware and can play VAG files but is very picky about the input format and wouldn't even accept the uncompressed WAVs created with Apple's Soundtrack. Awave is shareware that allows you to do batch conversions between many formats, however the free version is limited to processing one file at a time. Should I finally write an app to automate creating files for Beats, this will be worth the investment.

x00| 56 41 47 70 Magic Number, "VAGp" in ASCII
x04| 01 00 00 00 Version Number. All the VAGs in Beats are version 1
x0C| 10 A3 02 00 Waveform length. All the loops should be the same size for obvious reasons. Samples need not apply.
x10| 44 AC 00 00 Sample rate. 44100
x32| This is a 16 byte internal file name. In the case of JAM_0002, the first file is P1P1_Drums1.wav. This is only used by the VAG for reference, though an easy to understand naming convention is used. In this case this is page 1 (Drums) pad 1 (Default, no directional keys pressed) loop 1 (X button).
x48| 16 byte 00 padding
x64| Waveform begins

When importing or exporting a VAG file made specifically for Beats it's important to flip the values at x04 x0C and x10. Little endian for Beats, big endian for MFAudio/Awave.

BJS Intro

I'm still mulling over how to best report on the BJS format, as it's very complicated and while I think my understanding of the format is correct thus far, there are still some unknown values. Also, because of the way the file is structured, I'm having a hard time wrapping my head around how one might write an app that will output a proper BJS.

This is a basic explanation, code examples will follow soon, hopefully after I have successfully hand-built a file that works:
The beginning of the file is quite simple; there's a magic number, a file name (e.g. dance2), the folder name (jam_0002), the tempo, time signature, and the number of VAG files that follow. Then the first VAG appears, and things get a little more complicated.
After the first VAG, and in between every VAG thereafter is a chunk of data that varies in both size and content each time. This is the biggest missing piece of the puzzle. Luckily there are some hints at the end of the file (see below) that might explain what everything is. It'll hopefully click once I go over it three or four times.

After all the VAG files and their bizarre end-data, begins the naming of all the above files and their instrument sets. Finally, there's a large collection of 12 byte chunks that follow, for the most part, the same pattern: A 4 byte ASCII ID, a 4 byte offset (little endian), and a 4 byte length (also little endian). For example, you might see 42 50 4D 30 (BPM0) 1C 00 00 00 04 00 00 00 which says "The BPM (tempo) is at x1C and is 4 bytes long." Going to this offset at will then yield a value (e.g. 140 in JAM_0002). The exception to this is when the ASCII ID is followed by FF FF FF FF, in which case the next number is the length of the following related data. For example, the very first ID is "BIFF". After this is the 4 byte set of FFs and then a value. This value is equal to the size of the rest of the footer, minus the last 4 bytes. The very last 4 bytes of the file is the offset of the BIFF ID.

Still follow me?

The ASCII names range from obvious (e.g. BPM, NAME) to somewhat understandable (e.g. VAGS is the size of the following VAG information, NVAG is the number of VAGs listed, VAGF is a VAG file) to mysterious (e.g. UINT). Luckily most of the unknown ones are unused, and are probably reserved for something else.

It seems like a simple enough database, but working in reverse with no information at hand is proving to be a lot more difficult than I thought. If this structure looks at all familiar to anyone, please let me know, as it will save a lot of frustrations all around. Note: despite the odd "BIFF" string, the file seems unrelated to the Microsoft Excel or BioWare BIFF formats.

BJS Footer

Sorry for the lack of updates, but I haven't had much time to work on this and I'm not even sure the best way of going about explaining the BJS format. So I'm just going to post my raw notes. It's a backwards way to work, but it's best if you read the footer in order to find the meaning of the values found in the BJS. Posting a list of offsets isn't practical as every file is different.

Footer seems to follow "ASCII identifier, offset, length" pattern:
BIFF FFs then value = length of rest of footer (except last 4 bytes)
SONG FFs then value = length till VAGS
SDFV Offset to 1st Magic Number
SID0 Offset to 2nd Magic Number
NAME Offset to Internal name (Length = chars + 1)
AUTH "JAM_000x" name (+1)
BPM0 Tempo (Remember: these are OFFSETS to values)
BPB0 ? Always =4 Time signature? (Beats Per Bar)

VAGS FFs then length of ALL the following VAG related data
NVAG Number of VAGs
|VAGF VAG offset, length
|VNBD Unknown value after waveform
|VBDT Chunk of really unknown data, lengths vary greatly
|VENC ?? Appears occasionally. (Though not in JAM_0002) Always =2?
|LPNB Unknown value before next "VAGp"
|_______repeat for n VAG files...

SMPS FFs then length of ALL the following sample data
NSMP Number of samples
|*(following will not exist if NSMP = 0)
|SNAM Sample Name (length =+ 1)
|SDAT Offset and length following name. Odd. I assume assigns samples to a track.
|_______repeat for n samples...
*

VERS ? (Always =4)
NBRS Number of Bars? (Always =4, could do more?)
CNTI ? (Always =2)

TRKS FFs then length of ALL the following Track data
NTRK Number of tracks (Pads)
|TRAK Track info offset and length (76!)
|Assigns VAGs to tracks. See trackfooterdata.txt
|_______repeat for n tracks

STKS FFs then length of following data (12)
NSTK ? (=0)

SSEV ? Same offset as below, but no length (?)
NSSE ? (=0)

SSE0 ? Same as below, no length
SSET ? Ditto
NSST Number of sample sets
*SST0 Sample set name (e.g. PERC.) Only if NSST > 0 Repeat for n)*

UINT ? Same as below, no length
UIS0 Unknown data before page data, odd length (65)
Follows a bizarre pattern, seems to be the same throughout all Jam BJS':
(Decimal) 5 0 1 0 1 1 1 2 1 2 3 1 3 4 1 4 0
Might be related to page order

NUIM Number of pages

|UIM0 Page info. Assigns tracks/samples to pads within a page
|See pagefooterdata.txt for detail.
|_______Repeat for n pages

STG0 FFs then length of following data
NSTG ? (Always =0?) Found 8 bytes before BIFF

VERE ? (Always =4?) Found right before BIFF

Final 4 bytes is the offset of the first string (BIFF)

trackfooterdata.txt (Data is from JAM_0002)

00 00 80 3F Magic Number? Is probably of some significance, but I haven't seen other values
00 00 00 3F ??? The rest of the tracks are pretty predictable; the left, right and middle ones share equal values across instruments. Maybe balance?
04 00 00 00 Number of VAGs per track (Always 4)
00 00 00 00 First byte is VAG (1), third byte is button (X)
00 00 00 00 00 00 00 00 Pad
01 00 01 00 (2nd VAG (Drums2), 2nd button ([]))
00 00 00 00 00 00 00 00 Pad
02 00 02 00 (3rd VAG (Drums3), 3rd button (O))
00 00 00 00 00 00 00 00 Pad
03 00 03 00 (4th VAG (Drums4), 4th button (/\))
00 00 00 00 00 00 00 00 00 00 00 00 Pad
6A 61 6D 5F 30 30 30 32 5F 74 31 00 "jam_0002_t1" Must include trailing 00

pagefooterdata.txt

Data per page will be longer/shorter depending on the amount of pads.
Example is dan1 from JAM_0002.BJS. It has 3 pads:
03 00 00 00 Number of Pads
00 00 00 00 Pad 0 (Printed as 1)
01 00 00 00 Type ID. Loops use 1, samples use 2.
00 00 00 00 Track 0 (Known as _t1)
01 00 00 00 Pad 1 (2)
01 00 00 00 ID
01 00 00 00 Track 1 (t2)
02 00 00 00 Pad 2 (3)
01 00 00 00 ID
02 00 00 00 Track 2 (t3)
64 61 6E 31 00 00 00 00 "dan1"(name) +1, padded

This is crk1 from JAM_000D.BJS. It has 4 pads, one is a sample pad:
04 00 00 00 Number of pads
00 00 00 00 Pad 0 (Printed as 1)
01 00 00 00 ID
00 00 00 00 Track 0 (known as _t1)
01 00 00 00 Pad 1 (2)
01 00 00 00 ID
01 00 00 00 Track 1 (t2)
02 00 00 00 Pad 2 (3)
01 00 00 00 ID
02 00 00 00 Track 2 (t3)
03 00 00 00 Pad 3 (4)
02 00 00 00 SAMPLE ID
00 00 00 00 Sample 0 (s1)
63 72 6B 31 00 00 00 00 "crk1"