Blender Live Shapes (018; 06.09.2009; blender, processing)

Blender Live Shape 1.0 from MyInventions on Vimeo.

Blender Live Shape (BLS) - Blender 3D object with real-time displacement modifier (similar to normal map) based on live grayscale video from webcam.

How it works?

Processing script capture webcam and create grayscale images at fixed frame rate. Every frame is fallowed by data bit sended to port COM10.
Null-modem emulator (Com0com) send data from COM10 input to COM11 output.
Blender python API script (or Blender Game Engine script) refresh object texture with new image just created by processing program (data from COM11 make synchronization).
The same script refresh object displacement modifier with new texture, and also refresh Blender 3D window view.
This live modified object I called Bender Live Shape.

Practice

Object refresh rate depend on computer hardware configuration, mesh density and captured image resolution. For my configuration:
4225 vertex mesh, image: 200x150pixels, refresh rate: 26fps
16641 vertex mesh, image: 200x150pixels, refresh rate: 22fps
Processing and Blender files must be in common folder.
You need virtual serial port pair made with Com0com (you can replace this communication using standard input/output or send data through text file).
Establishing captured image resolution and vertex density remember, that number of vertex shouldn't by less than number of image pixels.
Start test of BLS performance with small Processing program refresh rate (set it in Capture functions parameters). If Blender gives no error - step it up.
Remember, that you can put GIFs and videos as Blender textures, so it's possible to use it instead of camera.
You should start Processing program before Blender script, but when finishing, first stop Blender and then Processing.

This concept files are full open:

Processing: BlenderLiveShape.pde
Blender editor version: BlenderLiveShape.blend
Blender Game Engine version: BlenderGameEngineLiveShape.blend

You could comment at BlenderNation:
http://www.blendernation.com/blender-live-shape-real-time-displacement-modifier/

Processing source code:

import processing.serial.*;
import processing.video.*;
Serial myPort;
Capture webcam;
int liczbaPikseli;
color Kolor;
int A, R, G, B, Y, j=1;
String name;
int t, T;

void setup() {
  size(200, 150);
  println(Serial.list());
  myPort = new Serial(this,"COM10", 9600);
  webcam = new Capture(this, width, height, 10); //framerate:10
  liczbaPikseli = webcam.width*webcam.height;
  //webcam.settings();
}

void draw(){
  if (webcam.available() == true){
    webcam.read();
    set(0, 0, webcam);
    loadPixels();
    for (int i=0; i < liczbaPikseli; i++){
      Kolor = pixels[i];
      R = (Kolor >> 16) & 0xFF;
      G = (Kolor >> 8) & 0xFF;
      B = Kolor & 0xFF;
      Y=int(0.3*R+0.59*G+0.11*B);
      A = 255 << 24;
      R = Y << 16;
      G = Y << 8;
      B=Y;
      pixels[i]=A|R|G|B;
    }
    updatePixels();
    name=str(j)+".png";
    saveFrame(name);
    myPort.write(char(j));
    j++;
    if (j==6){
      //T=millis()-t;
      //println(5000.0/T); //print real frame rate
      //t=millis();
      j=1;
    }
  }
}

Blender python API script:

import Blender
from Blender import Image
from Blender import Texture
from Blender import Material
from Blender import Object
from Blender import Window
from Blender import Modifier
from Blender import Scene
import serial
myPort = serial.Serial('COM11', 9600)

texture=Texture.Get('Textura')
texture.setType('Image')
scene=Scene.getCurrent()
plane=Object.Get('Plane')
modifier=plane.modifiers[0] 
myPort.flushInput()

for i in range(1, 301):
	y = ord(myPort.read(size=1))
	name=str(y)+".png"
	image=Image.Load(name)
	texture.image=image
	modifier[Modifier.Settings.TEXTURE] = texture
	Window.EditMode(1)
	Window.EditMode(0)
	Window.Redraw()

Blender Game Engine python script:

import Blender
from Blender import Image
from Blender import Texture
from Blender import Material
from Blender import Object
from Blender import Window
from Blender import Modifier
from Blender import Scene
import serial
myPort = serial.Serial('COM11', 9600)
texture=Texture.Get('Textura')
texture.setType('Image')
scene=Scene.getCurrent()
plane=Object.Get('Plane')
modifier=plane.modifiers[0] 

y = ord(myPort.read(size=1))
name=str(y)+".png"
image=Image.Load(name)
texture.image=image
modifier[Modifier.Settings.TEXTURE] = texture

controller = GameLogic.getCurrentController()
obj = controller.owner
mesh = obj.meshes[0]
obj.replaceMesh(mesh)