I see your point, However the majority of Android devices have lots of closed source firmware/drivers, a large part of the Android OS doesn't run without them.
I'm refering to Bootloaders, TrustZone OS, Trusted Applications, then firmware for Bluetooth, Wifi, GPU, Sensors and power management. All of those are always closed source binary blobs running in the background.
Difficult to say at the moment, but there might be some speculation as to why.
One might be that they dont security patches reverse engineered and vulnerabilities to come out faster (some critical and high severity fixes in GPU drivers/bootloaders were often delayed), and this decision was done long before LLMs were considered powerful/useful for vulnerability research.
Second reason might be simply "Control of the android ecosystem", Google may just want to build a wall around android and make it frustrating for other vendors to compete.
Third, again, this is only speculation, is the current push for electronic ID in the EU and other countries, as well as DRM/copyright protections for media and locally ran LLMs (we've heard of how google pushed small LLMs with chrome updates), if they can do the same on some high end android devices, Google would be more invested into further locking down Android devices.
It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.
That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?
Unfortunately I checked, SLI doesnt work for this situation. Because the program loading the LLM uses CUDA library, which doesnt account for/takes advantage of SLI for this purpose at least.
Correct me if I'm wrong, this is not the right way to ask this question.
LLMs are good at pattern recognition, so its less a type of math that they'll be good at, and more that when you provide documentation or text that can be easily parsed/compared to its training data/reasoning ability, the better answers you get from an LLM.
Also, you need to be knowledgeable at the same thing you are asking the LLM to do, to verify the answer it gives you (at least for the time being).
The current leap in research and its byproduct like ChatGPT are impressive (in the sense that it can pick up human text/speech and return back answers). But its not in any way a replacement for good Software Engineers, your job is safe (what is being posted on social media and news is extreme hype).
Here's the funny thing, if you ask people who interact with chat bots in any given service industry (banks, online orders...etc) they'll tell you how bad these things are, there is just no replacement for actual human beings with real life intuition (this kind of AI break through hasn't happened yet).
For the time being if you take time to specialize in a domain knowledge skill (physics, math, economy, or even agriculture) you'll be able to use the skills you learn from your CS degree in that domain and prosper as a result.
Here is the thing about pursuing a degree, it is a tedious awful long task. It doesnt actually look like the actual job you'll be doing when you finish this degree.
I'll go against the common advice and say, stick to it, finish the degree because it will give you good social mobility (when trying to travel for work to any country in the world, you'll either need to be a specialist in your field with few years experience or have at minimum a bachelor degree).
None of the modern chatbots that banks and online orders use are LLM based they are basically all low tech pattern matchers where they are given phrases to match on (utterances) with placeholders for the variable parts (slots) and then match those to intents. It’s in no shape form or fashion AI and accept for the voice to text part is 1960s technology.
>I'll go against the common advice and say, stick to it, finish the degree because it will give you good social mobility (when trying to travel for work to any country in the world, you'll either need to be a specialist in your field with few years experience or have at minimum a bachelor degree).
I don't like being a salaried employee (nor do I plan to become one, long term), but I might just finish my degree because life happens and there may be a time when I'll have to search for a job like that one day. So, purely as a backup option and because the sheepskin effect is very real
Hexrays used to be difficult to deal with if you want to purchase IDA Pro for the first time, due to their software getting leaked online.
They have eased the procedure to buy from them, but from time to time they'll ask you to fill out your info with national ID/passport (they say its because they don't want to sell their software to individuals under sanctions). This is despite them being based in Belgium (not the US).
For any serious work IDA Pro is highly suitable (the customization and scripting, loader examples and processor plugins...etc), on the other hand for side projects and basic security research Binary ninja and ghidra can go along way.
Given that this is an open source project and sniffing bluetooth can be fun but expensive (only some specialized tools can offer the best results or features), I feel that this project is much appreciated, came close to help those doing any research of bluetooth devices.
Video of the project's author introducing the tool and pros/cons of sniffing bluetooth traffic.
I'm refering to Bootloaders, TrustZone OS, Trusted Applications, then firmware for Bluetooth, Wifi, GPU, Sensors and power management. All of those are always closed source binary blobs running in the background.